Framework For Program Evaluation Research Articles

Building Healthy, Equitable, and Resilient Communities: Lessons Learned From Multisector Community Partnerships Addressing the Social Determinants of Health.

Multisector community partnerships play a pivotal role in public health strategy for understanding and addressing social determinants of health (SDOH) and promoting health equity. The work of these partnerships is well aligned with building community resilience. The Year 2 Improving Social Determinants of Health-Getting Further Faster (GFF) retrospective evaluation was designed to gather practice-based evidence that can inform future community-driven partnerships to improve chronic disease outcomes and advance health equity by addressing social determinants of health (SDOH). This paper sought to determine how the efforts of multisector community partnerships could also support community resilience. Guided by CDC's Framework for Program Evaluation in Public Health and health equity principles, the GFF evaluation engaged 14 Multisector Community Partnerships to better understand how their SDOH interventions contribute to community changes that improve chronic diseases and advance health equity. Data collection included qualitative discussions along with additional quantitative analyses of SDOH outcomes, costs, and prevention impacts. This paper focuses on insights gathered from qualitative discussions (2 virtual 60-minute discussions with members of each of the 14 Partnerships) around community resilience. Results from qualitative discussions demonstrate how Partnerships' SDOH efforts created parallel opportunities for nurturing community resilience. For example, Partnerships engaged community members in the design and implementation of SDOH interventions in ways that nurtured a sense of belonging and empowerment to shape their community. To further build community resilience, Partnerships mentioned the importance of sustained and flexible funding, help building stronger partnerships, and a shared goal to build resilience and engage community members. Findings indicate that community partnerships working to address SDOH can drive reciprocal improvements in community resilience-contributions that are critical for advancing health equity. Partnerships suggested that health departments may have opportunities to enhance community resilience through their role as funder, facilitator, and convener.

Using Intervention Mapping to co-design a psychosocial service with youth experiencing mental illness

BackgroundDescriptions of service development processes in the youth mental health sector are lacking. Further, youth with lived experience of mental illness are rarely involved in service design. Intervention Mapping (IM) is a well-established framework for program development, implementation and evaluation, yet its applicability in the youth mental health sector is unknown. This paper describes the use of IM methodology to co-design and develop a psychosocial service to support youth aged 10-25 years experiencing mental illness in Tasmania, Australia. MethodsThe six steps of IM were followed: 1) needs assessment; 2) define program outcomes and objectives; 3) program design; 4) program production; 5) implementation planning; and 6) evaluation planning. ResultsKey outputs of each IM step are described. The service was successfully co-designed with young people at the centre of each step. The service includes wrap-around psychosocial support from lived-experience peer-support workers; outreach; and flexible frequency/intensity/duration to achieve young people’s psychosocial goals. ConclusionsThis is the first study to document the use of IM in co-designing a psychosocial service with youth experiencing mental illness. IM may provide a valuable roadmap for the youth mental health sector in supporting collaborative service design, implementation and evaluation planning, and systematic documentation of service development.

CDC Program Evaluation Framework, 2024.

Program evaluation is a critical tool for understanding and improving organizational activities and systems. This report updates the 1999 CDC Framework for Program Evaluation in Public Health (CDC. Framework for program evaluation in public health. MMWR Recomm Rep 1999;48[No. RR-11];1-40) by integrating major advancements in the fields of evaluation and public health, lessons learned from practical applications of the original framework, and current Federal agency policies and practices. A practical, nonprescriptive tool, the updated 2024 framework is designed to summarize and organize essential elements of program evaluation, and can be applied at any level from individual programs to broader systems by novices and experts for planning and implementing an evaluation. Although many of the key aspects from the 1999 framework remain, certain key differences exist. For example, this updated framework also includes six steps that describe the general process of evaluation planning and implementation, but some content and step names have changed (e.g., the first step has been renamed Assess context). The standards for high-quality evaluation remain central to the framework, although they have been updated to the five Federal evaluation standards. The most substantial change from the 1999 framework is the addition of three cross-cutting actions that are core tenets to incorporate within each evaluation step: engage collaboratively, advance equity, and learn from and use insights. The 2024 framework provides a guide for designing and conducting evaluation across many topics within and outside of public health that anyone involved in program evaluation efforts can use alone or in conjunction with other evaluation approaches, tools, or methods to build evidence, understand programs, and refine evidence-based decision-making to improve all program outcomes.

Learning from educators: Implementation of a positive behavior support program targeting challenging behavior in children with autism

Challenging behaviors (CB) are a frequent co-occurring problem in children with autism spectrum disorder (ASD) and hinder their response to recommended interventions such as early intensive behavioral intervention (EIBI). The Prevent-Teach-Reinforce for young children (PTR-YC) program was implemented to meet community-based EIBI educators’ training and support needs in managing CB in their day-to-day work with families. Although this positive behavior support program has a strong empirical basis, its implementation by community-based educators has yet to be assessed from a systematic and structured program evaluation perspective. Using Chen’s (2015) theoretical framework for program evaluation, this study assessed the quality of implementation of PTR-YC as perceived by 17 educators who received training and supervision on applying PTR-YC among families of children with ASD receiving EIBI services. Educators’ post-intervention interviews and questionnaires were analyzed using the logical model for program evaluation to identify obstacles and facilitators to the implementation of PTR-YC.

Evaluation Resources for Asthma Programs and Beyond.

Evaluation can ensure the quality of public health programs. Systematic efforts to identify and fully engage everyone involved with or affected by a program can provide critical information about asthma programs and the broader environment in which they operate. To assist evaluators working at programs funded by the Centers for Disease Control and Prevention (CDC's) National Asthma Control Program (NACP), we developed a package of tools that build on the CDC's 1999 Framework for Program Evaluation in Public Health. The resulting suite of evaluation tools guides evaluators through a structured but flexible process, engaging a diverse array of interest holders and actively involving them in evaluation planning and implementation, all while strengthening their capacity to meaningfully contribute to the evaluation process. For our newest tool, our team reviewed the recent evaluation literature to create an enhanced version of the 1999 framework that describes important elements of professional evaluation practice. Although the original framework describes the steps to take in conducting an evaluation and the standards for a high-quality evaluation, our enhanced framework includes an explanation of how evaluators should approach their work: by incorporating critical reflection, interpersonal competence, situational awareness, and cultural responsiveness. In this article, we highlight many of the evaluation resources our team has created since the NACP's inception, culminating in a free e-text called Planting the Seeds of High-Quality Program Evaluation in Public Health. Public health professionals working in many types of programs - not just asthma - may find these resources useful.

Evaluating Student Equity Initiatives: A Student-Centred Approach

The robust evaluation of student equity initiatives is now an expected activity in Australian universities. This paper reports on the development of Swinburne’s Higher Education Participation and Partnerships Program (HEPPP) Evaluation Framework in which the project team adopted a student-centred design approach. The needs of Aboriginal and Torres Strait Islander students were positioned at the heart of the methodology and provided the starting point of explorations of student experiences in interviews and focus groups. We found that psychosocial support activities were critical to Indigenous students and those from low socioeconomic statue and/or regional or remote backgrounds and identified several additional Outcome factors of equity programs. Furthermore, we propose ‘Prosperity’ as a new Impact factor for evaluating equity programs to capture a broader notion of ‘success’. We argue that insights from existing institutional evaluation frameworks need to complement the Student Equity in Higher Education Evaluation Framework (SEHEEF) as Australia’s national framework and inform its continuous improvement.

Five ways to get a grip on applying a program evaluation model in health professions education academies

The proliferation of health professions educator academies across Canada and the United States illustrates the value they hold for faculty and institutions. Yet, establishing and evaluating the efficacy of them through program evaluation can be challenging. Moreover, academy leadership often lack the time, bandwidth skillset and personnel to undertake rigorous program evaluation efforts. We outline a step-by-step guide for getting a grip on evaluating health professions educator academies. Developing a plan for program evaluation in advance of any new academy initiative helps to ensure the academy calibrates and re-calibrates to accomplish outcomes and meet stakeholder expectations. It also provides a mechanism for tracking academy impact, which strengthens requests for funding, promotes sustainability and encourages continued buy-in and support from institutional stakeholders. For all of these reasons, we present the following recommendations: apply the relevant program evaluation framework(s); identify resources for program evaluation; prepare to tell your academy’s story; list desired program outcomes; establish a data collection plan; and obtain institutional review board approval.

Adopting the RE-AIM analytic framework for rural program evaluation: experiences from the Advance Care Planning via Group Visits (ACP-GV) national evaluation.

To support rigorous evaluation across a national portfolio of grants, the United States Department of Veterans Affairs (VA) Office of Rural Health (ORH) adopted an analytic framework to guide their grantees' evaluation of initiatives that reach rural veterans and to standardize the reporting of outcomes and impacts. Advance Care Planning via Group Visits (ACP-GV), one of ORH's Enterprise-Wide Initiatives, also followed the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. ACP-GV is a national patient-centered intervention delivered in a large, veterans integrated healthcare system. This manuscript describes how RE-AIM was used to evaluate this national program and lessons learned from ORH's annual reporting feedback to ACP-GV on their use of the framework to describe evaluation impacts. We used patient, provider, and site-level administrative health care data from the VA Corporate Data Warehouse and national program management databases for federal fiscal years (FY) spanning October 1, 2018-September 30, 2023. Measures included cumulative and past FY metrics developed to assess program impacts. RE-AIM constructs included the following cumulative and annual program evaluation results. ACP-GV reached 54,167 unique veterans, including 19,032 unique rural veterans between FY 2018 to FY 2023. During FY 2023, implementation adherence to the ACP-GV model was noted in 91.7% of program completers, with 55% of these completers reporting a knowledge increase and 14% reporting a substantial knowledge increase (effectiveness). As of FY 2023, 66 ACP-GV sites were active, and 1,556 VA staff were trained in the intervention (adoption). Of the 66 active sites in FY 2023, 27 were sites previously funded by ORH and continued to offer ACP-GV after the conclusion of three years of seed funding (maintenance). Lessons learned developing RE-AIM metrics collaboratively with program developers, implementers, and evaluators allowed for a balance of clinical and scientific input in decision-making, while the ORH annual reporting feedback provided specificity and emphasis for including both cumulative, annual, and rural specific metrics. ACP-GV's use of RE-AIM metrics is a key step towards improving rural veteran health outcomes and describing real world program impacts.

Analysis of a Bridging Program for Incoming First-Year Students to Transition from Secondary Education to Engineering Education

Bridging programs exist in many institutions to address the challenges incoming students experience. This study aims to identify some of the benefits and challenges of one such program at the Faculty of Applied Science and Engineering at the University of Toronto. Developed during the pandemic, this program aims to reduce the step-in height as students transition from secondary to post-secondary. This exploratory research uses a program evaluation framework to analyze this program. Literature review suggests a systems approach is an effective method, as it dissects the program into its fundamental parts and provides insights to the decision-makers of such programs [1]. We gather anecdotes from entities within the program and pair them with the analysis to identify if this program improves the student experience, and to what degree. This study is not an evaluation of the program, but rather a discussion about bridging programs and factors to consider when designing them. Findings from this study inform developers of bridging programs about their effect on the student experience.

Theory of change framework for program evaluation in Health Professional Education.

doi: https://doi.org/10.12669/pjms.40.4.8960
 How to cite this: Farrukh K. Theory of change framework for program evaluation in Health Professional Education. Pak J Med Sci. 2024;40(4):793. doi: https://doi.org/10.12669/pjms.40.4.8960

Our Ways, Your Ways, Both Ways - a multi-disciplinary collaboration to develop, embed and evaluate a model of social and emotional wellbeing care for Aboriginal and Torres Strait Islander young people who experience detention - Phase 1.

The National Strategic Framework for Aboriginal and Torres Strait Islander Peoples' Mental Health and Social and Emotional Wellbeing identifies building a strong Aboriginal and Torres Strait Islander led evidence-base to inform care as a key priority. Aboriginal and/or Torres Strait Islander adolescents in contact with the criminal justice system are a highly vulnerable group of Australians, with substantial unmet needs. There is limited evidence to inform culturally appropriate models of care that meet the social and emotional wellbeing needs of justice-involved Aboriginal and/or Torres Strait Islander adolescents. This project aims to develop, implement and evaluate an in-reach and community transitional model of social and emotional wellbeing care for Aboriginal and/or Torres Strait Islander adolescents (10-17 years old) who experience detention through close engagement with Aboriginal and/or Torres Strait Islander youth, Elders, researchers, practitioners and community members, and by drawing on culturally informed practice and knowledge systems. The project is based on a multi-level mixed methods design, with a strong focus on ongoing project evaluation (based on the Ngaa-bi-nya framework) and co-design. Co-design is facilitated through culturally safe and trauma informed participatory processes based on development of strong partnerships from project initiative, design, implementation and evaluation. Application of the landscape domain of the Ngaa-bi-nya framework for Aboriginal and Torres Strait Islander program evaluation will be explored in Phase one. Aboriginal and Torres Strait Islander adolescents with experience in detention will be engaged through one-on-one interviews with data collection through the Growth and Empowerment Measure (GEM) Youth (which will be adapted from the adult version and validated as part of this study), the Kessler Psychological Distress Scale (K-10), questions around alcohol and drug use, and narrative interviews exploring experience. Qualitative data will be analyzed using an inductive thematic approach, structured within the framework of the Ngaa-bi-nya landscape prompts. Quantitative data will be analyzed using descriptive statistics to provide a profile of the cohort. Findings from Phase one will be used to inform the development of a model of social and emotional wellbeing care that will be implemented and evaluated in Phase two.

Let the Program Evaluation Committee SOAR: Applying the Strengths, Opportunities, Aspirations, and Results (SOAR) Framework for Program Evaluation.

Annual program evaluations are important activities of all graduate medical education programs. Although the Accreditation Council for Graduate Medical Education provides general guidelines, there is substantial scope for educational innovation. Strengths, opportunities, aspirations, and results (SOAR) is a strengths-based framework for strategic planning. Because SOAR emphasizes positivity and engagement, it is an appealing framework for evaluating graduate medical education programs. Our objective was to demonstrate the feasibility and acceptability of SOAR in a program evaluation committee of a fellowship program to generate strategic initiatives. The authors used the four steps of SOAR within the program evaluation committee in 2022. Interviewers collected positive stories to understand program strengths. Then, rapid ideation was used to translate strengths into opportunities. These opportunities were condensed and refined for fellows to assess how well they align with aspirations. The ones that aligned best with aspirations were prioritized for implementation. Results were monitored through a scorecard based on specific, measurable, achievable, relevant, and time-bound (SMART) goals every month. Of 15 divisional members, 11 participated (73.3%). Five major strengths were identified: supportive environment, variety of cases, scheduling flexibility, integration with larger networks, and multidisciplinary collaboration. These 5 yielded 15 opportunities, which were refined and condensed to 9. Four were selected for implementation: scholarly works accountability group, hybrid-flex curriculum, fellowship weekly huddles, and structured electives. Scorecards have shown successful implementation during a 4-month period. SOAR is an innovative and feasible approach to program evaluation that uses trainee engagement to translate and synergize existing program strengths into actionable program improvement.

Reflections on leadership preparation research and current directions

This reflection addresses the need for research on how leadership preparation features develop candidates’ leadership skills and practices, as aligned to recent research on how principals best influence student learning. It reviews the nature of leadership preparation research, the investments in preparation programs, how the field has promoted leadership preparation research, and new developments in related research. Guskey’s program evaluation framework—which emphasizes evaluating the effects of professional learning on what candidates learn and do and the impact on their organizations—is useful in highlighting current shortcomings in how preparation features have been evaluated and identifying areas for further, more strategic research.

Evaluation of Centers for Disease Control and Prevention's Evaluation Fellowship Program.

The Centers for Disease Control and Prevention's (CDC's) Evaluation Fellowship Program is a 2-year fellowship that includes training, placement with a CDC program, and professional development funds. We evaluated whether the program contributed to CDC's evaluation capacity, prepared fellows for evaluation work, and contributed to their career advancement during its first 10 years. We used a mixed-methods approach, including conducting an online survey and telephone interviews. External evaluators sent surveys to all 152 alumni and all 123 mentors who participated in the program from 2011 through 2020 (first 8 cohorts) and interviewed 9 mentors and 15 alumni. A total of 110 alumni (72.4%) and 44 mentors (35.8%) completed surveys. Of 44 mentors, most agreed their fellow(s) contributed to their program's overall evaluation capacity (90.9%) and its ability to do more evaluation (88.6%). Most (84.2%-88.1%) alumni agreed that the Evaluation Fellowship Program prepared them to apply the 6 skill sets that aligned with CDC's Framework for Program Evaluation in Public Health. Support from the Fellowship office was significantly and positively correlated with performing evaluation tasks (β = 0.25; P = .004) and alumni obtaining their first job (β = 0.36; P < .001). Host program mentoring was significantly correlated with performing evaluation tasks (β = 0.27; P = .02) and alumni obtaining their first job (β = 0.34; P = .007). CDC's Evaluation Fellowship Program has made progress toward building CDC's evaluation capacity and preparing a public health workforce to use evaluation skills in various settings. A service-learning model that provides training and applied experiences could prepare a workforce to build evaluation capacity.

Returning to Normal Life after Deployment: What Can Emergency Response Organizations Learn from the Military?

Introduction:Emergency service workers are confronted with serious risks for their health, well-being and functioning. In order to prevent consequences to them and their families, emergency organizations should provide optimal support after an intensive period of employment. In many countries, the military pays special attention to the transition of their personnel from deployment to home via post-deployment adaptation programs (PDAPs). The objective of this presentation is to provide a structured analysis of the military approach to post-deployment adaptation and to identify potential lessons for emergency services.Method:A systematic literature search was performed to find original peer-reviewed studies on PDAP in six databases (MEDLINE, Embase, PsycINFO, Cochrane Central Register of Controlled Trials, PTSDPubs, and OpenGrey). The overall risk of bias of the articles was assessed using GRADE guidelines. The literature was analyzed guided by a program evaluation framework entailing different domains.Results:The search resulted in 1535 unique records that were screened for eligibility; 16 articles were included, of which only three showed low risk of bias. Most articles describe some form of third location decompression (N = 10) and also some agreement exists on how to adapt skills and cognitions after deployment (Battlemind; N = 4). The results suggest positive mental health effects and satisfaction of these elements.Conclusion:Empirical, high-quality evidence for PDAP is scarce. In addition, the existing literature reveals a lack of systematic method in describing the goals of PDAP and the ways of achieving these. Nevertheless, this study reveals promising elements that are in line with international guidelines, such as minimizing the level of exposure, intervention delivery and adjustment issues. We discuss how future research should incorporate these elements using a systematic approach.

What do we know about how the Program Evaluation Standards are used in public health?

Background: Released by the Centers for Disease Control and Prevention (CDC), Framework for Program Evaluation in Public Health prominently features the program evaluation standards (1999). The program evaluation standards (PES) include 30 statements in five domains: utility, feasibility, propriety, accuracy, and evaluation accountability. Despite decades of attention to the PES among framework users and others, how public health professionals apply these standards in their work is not well understood. Purpose: The study sought to identify notable commonalities in how the PES are used in public health. Setting: Application of the PES in evaluative work in public health and allied fields. Intervention: Not applicable. Research Design: The study included a search of subscription and nonsubscription sources to identify documents that included explicit content concerning use of standards in evaluative work in public health. Documents identified were screened using predetermined criteria to include or exclude each item in the study. Items included were reviewed and coded using codes developed before examining all documents. For each code, reviewers discussed data from all documents to identify commonalities and variations in application of standards. Findings: The literature search returned 405 documents to be screened (179 from subscription and 226 from nonsubscription sources). Thirty-eight items were included in the study based on initial screening (11 from subscription and 27 from nonsubscription sources). The study revealed that authors discussed standards as a regular component of evaluation work, but precisely how standards were used was not always explained in detail. Also, authors did not always discuss standards statements but sometimes solely focused on general domains (e.g., feasibility or accuracy). When authors discussed specific statements, they were more descriptive in how they applied the PES (i.e., compared with articles that focused on general domains). Overall, authors placed far greater emphasis on Accuracy and Utility standards, compared with Propriety, Evaluation Accountability, or Feasibility. In many cases, authors used the PES in combination with other resources (e.g., checklists, guidelines, or other standards). Although program evaluation is crucial to public health practice, the mechanics of how professionals consider, integrate, or use evaluation standards is not fully understood. Keywords: program evaluation; program evaluation standards; public health

Leadership in PhD (LeaP): A longitudinal leadership skill building program for underrepresented biomedical research trainees

Increasing diversity in the biomedical research workforce is a critical national issue. Particularly concerning is the lack of representation at more advanced career stages/in leadership positions. While there are numerous institutional initiatives promoting professional research skills (i.e., grant writing, presenting, networking) for underrepresented (UR) PhD trainees, there are comparatively fewer opportunities for leadership development. We present a blueprint for Leadership in PhD (LeaP), a cohort-based program aiming to equip UR biomedical research trainees with skills to succeed as academic, industry, and community leaders. In contrast to intensive short-term programs or workshops, LeaP is a longitudinal 4-year experience with an blend of didactic, self-directed, and experiential learning. First year trainees receive foundational didactic instruction on core leadership concepts coupled with facilitated peer discussions and one-on-one coaching support. We outline a program evaluation framework that assesses student learning, satisfaction, and program efficacy. Evaluation data from the inaugural year is presented and discussed.

Investigating the impacts of a preservice agriculture teacher recruitment program using Kirkpatrick’s program evaluation model

Although post-secondary school agricultural education degree programs strive to meet the need of supplying agriculture teachers, the gap between teacher demand and supply continues to exist. The Teach Ag Campaign is a joint effort of the agricultural education profession to recruit and retain agricultural educators and encourage students to pursue a career in teaching agriculture. Given the systematic program evaluation of the effects of Teach Ag Campaign efforts on agriculture teacher supply is lacking, this study investigated the outcomes and effectiveness of a preservice agriculture teacher recruitment program using Kirkpatrick’s program evaluation framework. The results indicated that the design and implementation of the Agricultural Education Institute (AEI) effectively met the needs of program participants, and the program positively affected participants’ knowledge, attitude, and aspiration regarding a career teaching agriculture. In addition, the findings demonstrated that the AEI contributes to producing agriculture teachers who support Florida's school-based agricultural education programs. Based on the results of this study, recommendations for practice in terms of participant recruitment, program design, implementation, and future research were proposed.

A Roadmap to a Comprehensive Evaluation of a Clinical Ladder Program

Clinical ladder programs were designed to support nurse retention and professional development. Hospital systems are vigorously working to recover from the critical staffing shortages exacerbated by the pandemic. The Centers for Disease Control and Prevention framework for program evaluation was utilized to assess a hospital professional nurse clinical ladder. These findings suggest favorable outcomes in commitment to stay and professional development. This evaluation can serve as a potential resource for organizations to evaluate and enhance programs to support nurse retention.

Therapeutic change in adolescents sexual offenders

IntroductionA number of studies have shown that the treatment of adolescents who have engaged in sexually abusive behavior (AESABs) reduces their risk of sexual recidivism. However, few studies have examined the internal processes leading to these results. ObjectiveThe purpose of our research was to determine whether the treatment of AESABs (n=43) influences the following factors related to sexual offending: deficient social skills, social isolation, ineffective coping strategies, and cognitive distortions. The impact of motivation for change and trauma symptoms on therapeutic progress was also examined. MethodPre/post-treatment analyses were completed. ResultsThe results of our pre/post-treatment analyses indicate significant changes in social isolation, coping strategies and cognitive distortions, and a significant reduction of trauma symptoms. Only social skills remained deficient. The data on motivation were not sufficient to establish whether motivation was linked to the progress of the subjects. DiscussionOur study highlights the importance of interventions targeting the acquisition of social skills in AESABs. Trauma is also an important factor to consider in the assessment of therapeutic progress, despite not being directly related to sexually abusive behaviour. The limits of the study highlight the importance of establishing an a priori program evaluation framework.