Program Evaluation Research Articles

Preregistration in Evaluation: Lessons About Preregistration of Outcomes and Analytic Protocols From a National Job Training Program Evaluation

Preregistration in Evaluation: Lessons About Preregistration of Outcomes and Analytic Protocols From a National Job Training Program Evaluation

Using the PEBIC criteria to assess the quality of responses provided during a Family Planning radio program

IntroductionRadio remains a dominant source of information, in Sub-Saharan Africa, with 75% of the population relying on it. In Nigeria, radio reaches over 70% of citizens and stands as a vital tool for behavior change. Tailoring radio programs to address needs and cultural contexts of communities has proved essential for building trust and influencing behavioral change. Traditional evaluation of radio interactive programs focus on listener engagement metrics but overlooks the evaluation of the quality of the resource person's responses to the caller's questions. This study aimed to assess the quality of information delivered by resource persons during the interactive segments of a Family Planning (FP) radio program using a criteria.MethodsA qualitative analysis was conducted on caller questions and resource person responses from FP radio programs in three Nigerian states. A customized ranking system based on “Presentation,” “Empathy,” “Provider's Bias elimination,” “Information correctness,” and “Context Specificity” (PEBIC) was used by FP experts to assess the resource person's response quality. The scores for the responses were segmented and categorized into low, moderate and high quality using the mean, and standard deviation then analyzed using SPSS version 20.ResultsFor the technical segment of the radio program, while individually assessing each item on the criteria, the quality of responses was high, with an average of 91.2%. For the entertainment education segment of the radio program, while individually assessing each item on the criteria, the quality of responses with empathy was low (28%), however, the quality of responses for other criteria was high, with an average of 79.8%. Furthermore, in the overall quality of response for the technical segment, the responses given by the resource persons were of high quality (81%) while those of the entertainment education segment were of moderate quality (58%).ConclusionThe radio program excelled in delivering technical information but fell short in fostering adequate empathy during the entertainment education segment. Additional training for resource persons to hone their skills in embedding their message delivery in an empathetic frame thus strengthening the emotive connection with the audience is key. This could significantly improve the program's overall impact.

Virtual Health Research Capacity Strengthening in Low- and Middle‑Income Countries: A Systematic Integrative Review.

Background: Effective and scalable strategies are needed to develop health research capacity in low- and middle‑income countries (LMICs). Health research capacity strengthening (HRCS) focuses on boosting production and utilization of health research, with clinicians as key target participants. Despite the increased prevalence of virtual HRCS programs, there has been no review of the evidence for those targeting LMIC clinicians to date. Objective: This review characterizes the use of virtual tools in HRCS programs for clinicians in LMICs and describes the impacts, facilitators, and barriers associated with these programs. Methods: Following our protocol (PROSPERO; CRD42020152510), we employed an integrative review methodology. We adapted Cooke's Research Capacity Development for Impact framework by adding "equity" as a new domain and used it to evaluate programmatic impacts. We retrieved relevant articles from five databases and gray literature. Included articles were extracted and stratified by degree of virtual delivery. We analyzed virtual tool usage via content analysis. Using NVivo, we coded until theoretical saturation and analyzed data using the constant comparison method. Findings: From 1397 articles, 58 met inclusion criteria. Most programs were hybrid, and e‑courses were the most used virtual tool. Articles described impacts across all framework domains; the most discussed were skills and confidence building. Facilitators included user‑friendly platforms, interactive content, and strategies to improve program access, including financial and technological support. Some programs incorporated hybrid strategies to foster trust among participants and virtual mentors. Barriers included a lack of or an unfavorable local research culture. Conclusions: Recommendations from this review may guide the design and implementation of virtual HRCS programs for LMIC clinicians. These include selecting well‑fitted program participants, intentionally designing program structure and content, conducting needs assessments or pilots, incorporating equity as a programmatic target, ensuring longitudinal program evaluation and monitoring, and utilizing a comprehensive conceptualization of program sustainability.

Maintaining Access to Education Amid Conflict: An Evaluation of Open and Remote Learning Programmes for Marginalised Communities in Wartime Sudan

This study looks at how open and remote learning affects underserved groups. It focuses on case studies and successful methods that show good examples of teaching communities that face discrimination or are not like most people. The goal is to review these examples and learn from them, like the most important things that affect how well open and remote learning works for these communities. It also helps us understand the difficulties these communities face and how to deal with them. The aim is to give clear understanding into making and carrying out open and remote learning programmes for these communities in ways that can last and make real change. In summary, the key results of the study show the programme improved access and achievements but more is needed on career preparation, closing socioeconomic gaps, bolstering technology skills training, and creating emergency-responsive solutions in the wartime environment.

Evaluation of a capacity-building program to enhance evidence-based practice of cognitive rehabilitation providers: a pilot study.

Cognitive rehabilitation is a complex and challenging area of brain injury rehabilitation practice, and specialized training programs are needed to upskill practitioners to ensure effective, evidence-based interventions. We developed a cognitive rehabilitation capacity-building program (CogCap) to address this need. The aims of this study were to pilot the CogCap program with a multidisciplinary group of cognitive rehabilitation providers; to evaluate whether it improved participants' self-ratings of knowledge, skills, and confidence in cognitive rehabilitation; and to explore their perceptions and experiences of the program. CogCap was evaluated in a pilot study with 12 cognitive rehabilitation providers, who included occupational therapists, speech pathologists, and psychologists working in brain injury rehabilitation. CogCap components included online self-paced learning modules, workshops, and expert masterclasses completed over a 3-month period. Measures included degree of participation in CogCap components; pre- and post-self-ratings of knowledge, skills, and confidence; and qualitative interviews of CogCap participants about their experience. There were high levels of engagement with the workshops and significant improvements, with large effect sizes in perceived knowledge (d = 0.81, p < 0.05), skills (d = 0.65, p < 0.05), and confidence (d = 1.30, p < 0.01). Four themes were derived from the qualitative interview data: (1) learning through the program, (2) engaging in the program, (3) enhancing practice, and (4) supporting future change. The pilot evaluation provides preliminary evidence of CogCap's effectiveness and acceptability to cognitive rehabilitation providers. Larger scale implementation and evaluation is recommended. http://links.lww.com/IJEBH/A334.

Analizando alternativas metodológicas para la investigación educativa: Design Science Research (DSR)

Educational research requires robust methodologies that aid in the design of new pedagogical practices. In this context, Design Science Research (DSR) presents itself as a solid alternative to Design-Based Research (DBR). The main objective of this article is to present DSR as a methodological tool for educational research, highlighting its foundations, variants, and practical applications. Throughout the article, we will conduct a detailed analysis of DSR and its variants through examples of the development of educational technologies, pedagogical innovations, and the evaluation of educational programs. The most notable results show that DSR, with its cyclical structure that includes the relevance cycle, the design and construction cycle, and the rigor cycle, has been successfully applied in the development of online learning platforms, formative assessment tools, and pedagogical models. In comparison, DBR is characterized by its collaborative approach and methodological flexibility, although it presents challenges in replicating results and selecting control groups. The conclusions indicate that DSR offers a structured and rigorous methodology that can improve educational research through the creation and evaluation of resulting products, ensuring greater replicability. On the other hand, DBR is valuable for its practical and collaborative approach; the choice between DSR and DBR will depend on the specific nature of the research and the available resources, suggesting that integrating both methodologies could offer a more comprehensive and effective solution to address educational challenges.

Increasing Social Connection in Permanent Supportive Housing: A Participatory Study on Therapeutic Recreation for People with Experiences of Homelessness.

Social exclusion and isolation due to homelessness are compounded in older people by physical frailty and stigma. Interventions to support older people with experiences of homelessness often neglect social dimensions of well-being. In this participatory action research study, residents, staff, and researchers collaborated to develop, implement, and evaluate therapeutic recreation programming in a permanent supportive housing facility for older people with experiences of homelessness (n=68). A community advisory group 'The Exchange,' co-designed program delivery and evaluation. Quantitative evaluation data consisted of participant demographics, participation rates, goals set and quality of life questionnaires at three points in the study. Qualitative data included co-design meeting notes (n=24), and interviews with residents (n=19) and staff (n=20). Quantitative data were reported descriptively, and qualitative data were analyzed thematically, using social and ecological theories of care from recreation studies as a sensitising lens. Data were synthesised to develop a final interpretation. Sixty-one (90%) residents participated in recreation programming. Residents set 253 goals; social goals were the most popular. Residents reported increased confidence to learn new skills, engaging in meaningful activities, and improved social connection with their community. Co-designing recreation programming contributed to an inclusive social environment that enhanced individual and community connectivity. The findings demonstrate the benefits of therapeutic recreation in promoting well-being among older people with experiences of homelessness living in permanent supportive housing.

Evaluating a Canadian tertiary care emergency department physician wellness program.

In response to increasing physician moral distress, the Department of Emergency Medicine at an academic, tertiary care hospital established a physician wellness program in 2018.We conducted a program evaluation at the 5-year mark which included the development of a program evaluation framework. We used a utilization-focused logic model for the wellness program as per Van Melle. We identified inputs-outputs-outcomes and analyzed multiple data sources. These sources included qualitative and quantitative feedback from physicians, participation data from wellness initiatives, exit surveys, and engagement in departmental committees. Over 5 years, the program periodically collected data on self-reported wellness of attending physicians along with participation data. In 2023, physicians ranked their occupational wellness at 5.93 on a 10-point Likert scale, with 58% highlighting supportive colleagues as their main protective factor and 42% identifying sequelae of emergency department crowding as a negative factor. Of survey respondents, 65% felt the program positively impacted their occupational wellness. Successful initiatives included public appreciation of physicians and creation of a "process map" to help guide physicians through a patient complaint. However, other initiatives such as wellness drop-in sessions were less successful. While social events were appreciated, initiatives focusing on addressing barriers to occupational wellness were valued the most. This program evaluation and logic model guided the planning committee to emphasize enhancing occupational wellness, recognizing physician accomplishments, and improving patient relations. It served as an operational example for the development and implementation of a wellness program within a large, tertiary care emergency department. Other centers can implement similar activities to create or further enhance their wellness programs.

Monitoring and adaptation of a system dynamics approach to prevent childhood overweight and obesity: findings from the LIKE programme

BackgroundThere are few examples of public health programmes rooted in system dynamics methodology. The aim of this paper was to broaden the evidence-base on the implementation and evaluation of a system dynamics programme for obesity prevention, using the Lifestyle Innovations based on youth’s Knowledge and Experience (LIKE) Programme as a case study. In LIKE, system dynamics principles were operationalized around three central pillars: the action programme is (1) rooted in a system-based understanding; (2) integrated in the local context and (3) dynamic.MethodsThis study took place in an urban setting in Amsterdam, the Netherlands, as part of the LIKE programme. The action programme consisted of establishing thematic action groups around previously identified leverage points within the system of overweight-related behaviours among adolescents. An action monitoring register was used to monitor action development and implementation, including the targeted system level. To track action implementation and adaptation over time, we conducted an in-depth evaluation using ripple effects mapping and additional interviews for three action groups. This data was analysed by performing a thematic content analysis.ResultsDuring the 6-year course of LIKE, 63 action ideas were formulated by 12 action groups, and 22 of these actions were implemented. Most of these implemented actions targeted lower system levels. A total of 9 of the 22 implemented actions were incorporated in existing initiatives. We observed that operationalization of system dynamics principles influenced the form of the action programme. Action ideas were dynamic in the sense that they changed over time or were abandoned because of growing system insights and/or factors within the wider context. This required shifting the focus from individual actions to the programme as a whole and formulating action ideas in terms of their function in changing the system, instead of on its form.ConclusionsUsing LIKE as a case study, this study provides an example of the output of a system dynamics action programme. We show how leverage points can be used as a starting point to develop action ideas that target lower and higher system levels. This demands monitoring and evaluation that facilitates continuous customization of the programme.

Establishing and Evaluating an Interdisciplinary Summer Undergraduate Research Training Program in Maternal, Child, and Family Health.

Representation of professionals from populations affected by health disparities is crucial for equality in healthcare practice and research. Undergraduate research programs focused on health disparities can deeply impact students' self-efficacy, interest, and persistence in health and research-focused careers. In this study, we designed, implemented, and evaluated a novel summer undergraduate research program focused on maternal, child, and family health. Over four summers, 45 students from a range of majors engaged in interdisciplinary faculty-mentored research projects, using methods ranging from molecular biology to community-based participatory research. The impact of the program on these students-the vast majority of whom were from historically marginalized communities-was demonstrated through an evaluation study including pre-program and post-program surveys and tracking post-program student outcomes, including co-authoring publications and conference presentations, and persisting in health and research fields after graduation (an outcome for 76% of our graduates). This program was designed to be accessible and supportive and to offer students the opportunity to engage in research projects focused on the health of the communities in which they are personally invested. Our program makes a novel contribution to the field of health disparities training programs, as we did not encounter in the literature any prior studies describing the establishment and evaluation of an undergraduate research program in maternal, child, and family health. The insights we gained from implementing this program could be utilized by other institutions seeking to design undergraduate research programs focused on maternal, child, and family health specifically, or on health disparities more broadly.

A Case Report of a Chair Yoga Intervention for Acute Geriatric Psychiatry

While recreational therapists (RT) report using nonpharmacological approaches to treatment, there is limited evidence-based practice described in RT related to the use of adapted yoga techniques in geriatric psychiatric settings (Wiles et al., 2021). The purpose of this case study is to describe an interdisciplinary team approach and the specific RT process used to assess, plan, implement, and evaluate a chair yoga program for adults in a geriatric psychiatric unit. The results of the physical outcome measures and anecdotal evidence to support the program are also presented. Significant findings were identified from pretest and posttest analysis of flexibility scores using the Senior Fitness Test (Rikli &amp; Jones, 1999). Program replication and evaluation would be beneficial in generalizing the effects of chair yoga.

Transmission in interprofessional humanitarian teamwork: a scoping review

ContextSince the end of the 1990s, the humanitarian sector lives a process of professionalization which has created some challenges for organizations and workers. These changes, which aim to standardize and harmonize practices, have also added issues of worker retention, creation of intercultural and interprofessional teamwork, skills development, health, and maintaining expertise in the fields.AimIdentify the mechanisms of knowledge transmission within humanitarian intervention collectives in a professionalization context.MethodThe design follows the five-step scoping analysis method proposed by Arksey and O’Malley (Int J Soc Res Methodol 8:19-32, 2005).ResultsThe study examines dynamics within humanitarian work groups, focusing on relations between international and national workers, professional segmentation, and collective resilience. It also explores essential knowledge in humanitarian work (contextual, technical, collaborative, metacognitive, and managerial) and the main modes of knowledge transmission: reflective practices, on-the-job transmission, and training. Understanding these interactions and transmissions optimizes effectiveness and skill development in humanitarian contexts.Discussion/conclusionThis article discusses professionalization through the rationalization of the humanitarian sector. There is a lack of scientific interest in knowledge transmission through work groups, with a focus on training programme evaluation. Existing dynamics on the ground is harder to control, hindering the evaluation of investment effectiveness. This trend is accompanied by a scarcity of field studies. In addition, there is a need for collective adaptation to respond to unusual situations. However, professionalization, subject to the constraints of donors, can create difficulties for non-governmental organizations (NGO) to establish conditions conducive to resilience or innovation. Our findings call on policy-makers to support adaptable frameworks that improve knowledge sharing and team dynamics in humanitarian response.

The Role of Disability Resources and Educational Supports Centre Towards the Inclusive Higher Education: A Case Study from the United States

ABSTRACT Given the growing number of inclusive higher education, it is critical to understand how the disability support centre works for college students with disabilities. The aim of this study was to explore disability support services for college students with disabilities in the United States and to provide implications for the Korean higher education system. We examined the perspectives of five specialists on the implications of disability support services on campus based on the mid-western public ‘A’ university in the United States via in-depth interviews, field notes, and program evaluation. Thematic analysis was used to identify the relevant theme: overall disability support services on campus, qualification and roles of specialists, and implications for individualised supports. By examining the successful case of disability support services at the ‘A’ university, it would provide implications for practices and policy to reform the college disability support centre in South Korea.

Peran Pelatihan dan Pengembangan Karyawan dalam Meningkatkan Kompetensi SDM di PT Timah Tbk

This study aims to analyze the role of employee training and development in improving human resource (HR) competency at PT Timah Tbk. Using a qualitative approach and literature study method, this study collected secondary data from various sources such as journals, articles, and company reports related to HR training. The results of the study indicate that the training program at PT Timah Tbk has a positive impact on improving employee technical and managerial skills. However, there are several obstacles that hinder the effectiveness of training, such as limited facilities and less than optimal evaluation. Therefore, the company needs to conduct continuous evaluation and improvement of the training program to achieve maximum results. This study provides insight into the importance of proper training management to improve HR competency in supporting the company's success.

Implementation and evaluation of the gender resilience, resistance, empowerment, and affirmation training (GREAT) pilot program.

This article reports program evaluation findings of the effectiveness of a pilot training program integrating key principles from the psychological framework for radical healing (French et al., 2020) and models aimed at decreasing internalized stigma (Israel et al., 2021). Our main goal of the training program was to increase psychotherapists' knowledge and skills related to competent and affirming practice with two-spirit, transgender, and nonbinary clients who are Black, Brown, and people of color adults. Pre- and posttraining qualitative and quantitative data were collected from 82 psychotherapist attendees to assess the acceptability of the training for learning goals and to provide preliminary evidence of training effectiveness for the training's learning objectives. Descriptive content analysis of qualitative data was used to assess the degree to which the training addressed psychotherapists' goals and increased knowledge and skills. Regression analyses of quantitative data found significant increases in psychotherapists' self-reported knowledge of key concepts and frameworks and confidence to apply new learning to psychotherapy practice. Program evaluation results largely support the acceptability and effectiveness of this training for increasing self-rated competency for psychotherapy practice with two-spirit transgender, and nonbinary Black, Brown, and people of color adult clients. Implications are discussed. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

A Three-Year Examination of U.S. Undergraduates Attending a High-Ranking Public College During COVID-19: Contributions to the Field’s Understanding of Psychological Adjustment

ABSTRACT This study focused on the psychological adjustment of college undergraduates during the COVID-19 pandemic. Participants attended a small public university in the mid-Atlantic region of the U.S. Using a cohort research design, data were collected across a three-academic-year period (N = 3323). Data included measures of change in functional adjustment to the pandemic (i.e., self-perception of anxiety, depression, loneliness, psychological distress, alcohol and substance use, academic performance), and standardized measures of anxiety, depression, personality, and childhood trauma. Student responses decreased in severity and intensity over the three-year period. Standardized anxiety scores remained in the mild range of severity. Females and students identifying as other than male and female, scored higher on all measures. Students endorsing childhood trauma scored higher on all measures. The authors offered recommended next steps in research, program evaluation, and clinical practice to help inform the effective management of future pandemics or natural disasters.

ReBreed21-ET: Evaluation of a rapid resynchronization program that allows timed embryo transfer every 21 days.

ReBreed21-ET: Evaluation of a rapid resynchronization program that allows timed embryo transfer every 21 days.

Investigation of Nanoflow Liquid Chromatography High-Resolution MS (nanoLC-HRMS) for the Analysis of Veterinary Drug Residues in Fish and Shrimp.

A wide-scope screening method for veterinary drugs and other contaminants in aquaculture products using LC high resolution MS (HRMS) was developed previously. Using nanoflow LC could significantly increase the sensitivity of this method. The objective of this study was to evaluate the potential advantages of nanoLC- HRMS as a screening method for veterinary drug residues in seafood. Preliminary nanoLC-HRMS investigations included comparison of nanospray source options and evaluation of nanoLC columns, mobile phases, and gradient programs. The optimized method was tested using tissue fortified with a mixture of veterinary drugs and pesticides. Using nanoLC columns with integrated emitters provided stable electrospray ionization resulting in chromatographic peaks with reproducible retention times and peak areas; diagnostic MS2 product ions were also detected. Coupling nanoLC to HRMS increased the area counts for many target analytes by 2-3 orders of magnitude. Less concentrated extracts could be analyzed with simplified preparation and minimal ion suppression. Replicate extractions of fortified fish generally gave relative standard deviations under 20%. Most analytes could be identified at concentrations ten-fold less than the target testing levels with detection limits in final extracts corresponding to 0.01-0.02 picograms injected on-column. Acceptable linearity was observed using solvent, matrix-matched and matrix-extracted standard curves. Disadvantages of nanoLC-HRMS included the inability to detect some classes of compounds, i.e., dyes and avermectins, and longer wait times between chromatographic analyses. Significant increases in sensitivity were observed with nanoLC-HRMS. Analytical results from fortified fish and shrimp showed the method to be suitable for qualitative screening, analyte identification, and quantification with minimal matrix effects. A wide variety of veterinary drugs and pesticides could be reproducibly detected and identified in fortified fish and shrimp using nanoLC-HRMS with minimal clean-up. Residues from imported laboratory samples were also detected and identified using this method.

Evaluation of a volunteer transportation program that supports age-friendly rural communities

Evaluation of a volunteer transportation program that supports age-friendly rural communities

Evaluation of cargo bike program for parcel deliveries in a medium-sized city

Evaluation of cargo bike program for parcel deliveries in a medium-sized city