Program Evaluation Research Articles

Evaluation of the Excellent Class Program in Improving the Quality of Education in Junior High Schools. Muhammadiyah - Bandung

This study applies the Context, Input, Process, Product (CIPP) program evaluation model with a descriptive qualitative approach. Data collection refers to the CIPP evaluation instrument table which is compiled based on predetermined criteria, through observation, interview, and documentation techniques. Data analysis involves data collection, data reduction, data presentation, drawing conclusions, and data verification. The results show 1. Evaluation of the program context in the number of students who have talents and creativity that are facilitated more optimally. The excellent class program to create superior students and shows that Muhammadiyah schools have quality students. 2. The Input Evaluation applied includes student screening, testing, willingness to join the excellent class. 3. Evaluation of the program performance process is not optimal. The relationship between the implementer and students is going well, with full trust from the parents of students regarding student development. 4. Evaluation of the program's product results showed good achievements, although not yet fully maximized. 80% of Muhammadiyah graduates successfully continued their education to favorite state schools. The superior class program to be continued with the condition of reformulating the initial objectives of the program.

A research agenda to support economic resilience in US oil and gas producing communities

Abstract Major reductions in greenhouse gas emissions will require a transition away from fossil fuels, including oil and natural gas. However, little research has examined the implications of such a transition for the workers and communities who depend on these industries to support local and regional economic wellbeing. In this perspective, we lay out a research agenda that can help inform policymakers as they seek to craft effective policies to support affected communities. We focus on the United States, the world’s largest oil and gas producer, and identify three key policy areas where new scholarship is needed to inform policymaking: economic and workforce development, public finances, and environmental remediation. Although it is not a comprehensive research agenda, we identify dozens of distinct research questions that will require a mix of methods and disciplinary lenses, including basic data gathering, community engagement, program evaluation, policy analysis, political analysis, and more. The goal of this article is to encourage scholars to take up these topics and expand them in the years ahead to ensure that oil and gas communities become more economically resilient in the face of deep uncertainty over the future of the domestic and global energy system.

Achievement of safer learning facilities standards in the disaster safe education unit program at Panggang elementary school

This study aims to discover and describe the achievement of safer learning facility standards for the disaster safe education unit program at Panggang elementary school, Bambanglipuro. This study is a descriptive evaluative research mixed methods approach with a sequential explanatory strategy. The sampling technique was carried out with saturated samples. Data collection techniques used observation, interviews, documentation studies, and questionnaires. Data validity uses internal validation, technical triangulation, and source triangulation. Quantitative data analysis uses a percentage formula calculation according to Sugiyono, while qualitative data analysis uses an interactive analysis model. However, each aspect still has gaps. The gaps that occur are not too significant, but still affect the achievement of safer learning facilities. Efforts are needed from Panggang elementary school Bambanglipuro to reduce or even eliminate existing gaps so that safe learning facilities in the disaster safe education unit program can be fully achieved. Keywords: program evaluation, disaster safe education unit program, safer learning facilities

Process evaluation of an mHealth-based school education program to reduce salt intake scaling up in China (EduSaltS): a mixed methods study using the RE-AIM framework

BackgroundAn mHealth-based school health education platform (EduSaltS) was promoted in real-world China to reduce salt intake among children and their families. This progress evaluation explores its implementation process and influencing factors using mixed methods.MethodsThe mixed-methods process evaluation employed the RE-AIM framework. Quantitative data were collected from a management website monitoring 54,435 third-grade students across two cities. Questionnaire surveys (n = 27,542) assessed pre- and post-education effectiveness. Mixed-effects models were used to control cluster effects. Qualitative interviews (23 individuals and 8 focus groups) identified program performance, facilitators, and barriers. Findings were triangulated using the RE-AIM framework.ResultsThe program achieved 100% participation among all the third-grade classes of the 208 invited primary schools, with a 97.7% registration rate among all the 54,435 families, indicating high "Reach." Qualitative interviews revealed positive engagement from children and parents through the "small hands leading big hands" strategy. The high completion rate of 84.9% for each health cloud lesson and the significant improvement in salt reduction knowledge and behaviors scores from 75.0 (95%CI: 74.7–75.3) to 80.9 (95%CI: 80.6–81.2) out of 100 demonstrated the "Effect" of EduSaltS. The program's "Adoption" and "Implementation" were supported by attractive materials, reduced workload via auto-delivered lessons/activities and performance evaluation, and high fidelity to recommended activities, with medians 3.0 (IQR: 2.0–8.0)/class and 9.0 (IQR: 5.0–14.0)/school. Stable course completion rates (79.4%-93.4%) over one year indicated promising "Maintenance." Apart from the facilitating features praised by the interviewees, government support was the basis for the scaling up of EduSaltS. Barriers included the lack of smartphone skills among some parents and competing priorities for schools. Unhealthy off-campus environments, such as excessive use of salt in pre-packaged and restaurant foods, also hindered salt reduction efforts. The program's scalability was evident through its integration into existing health education, engagement of local governments and adaptation across various mobile devices.ConclusionsThe mHealth-based school health education program is scalable and effective for public salt reduction in China. Identified barriers and facilitators can inform future health program scale-ups. The program's successful implementation demonstrates its potential for broader application in public health initiatives aimed at reducing dietary salt intake.

A Survivor-Centered Framework for Domestic Violence Brief Crisis Intervention: A Stakeholder-Informed Approach

Program evaluation and assessment are critical for understanding the impact of domestic violence services and interventions and for making necessary refinements and revisions to programs. Demonstrating the impact of domestic violence programs (e.g., hotlines and shelters) poses challenges due to the complexities associated with abuse experiences. The use of an assessment framework will help categorize users’ multiple needs, clarify specific components of intervention approaches, and ensure that services meet clients’ needs. This article provides a comprehensive overview of a theoretical and stakeholder-informed approach used to develop an assessment framework for the National Domestic Violence Hotline (The Hotline). The research resulted in a framework which underscores the importance of survivor empowerment and support as key survivor outcomes. This article presents detailed information about the SAF-T project including methods and processes used to develop the survivor-centered framework and provides guidance for those interested in developing similar theoretical and stakeholder-informed frameworks for program planning, measuring performance, and evaluation efforts.

Integrating Data Analytics Training into Curricula at Small Colleges: A Practical Framework for Enhancing Student Skills

Quantitative Data Analytics (QDA) is a critical skill for professionals across various natural science disciplines. However, college curricula, especially at small colleges, often lag behind the evolving workforce’s needs due to limited faculty resources. Despite abundant training materials, there is a noticeable gap in practical frameworks for effectively developing and implementing QDA training programs, particularly addressing existing curriculum gaps and resource constraints. This study presents a practical framework for developing introductory QDA training programs tailored for non-statistical majors at resource-constrained small colleges with various educational settings. Employing a design science strategy, we developed a modular co-curricular QDA Training Program (CQDATP), consisting of five training modules refined over three iterative phases. This flexible design integrates seamlessly with existing courses or can function as standalone workshops, bolstering students’ preparedness for advanced courses and research opportunities. Our program evaluation indicates that our approach significantly enhanced students’ QDA capabilities without necessitating extensive curriculum overhauls. By focusing on adaptability and resource efficiency, our framework holds the potential to benefit educators, curriculum developers, and administrators at small colleges, ensuring that graduates are well-prepared for data-driven industries, even when resources are limited.

Key Improvements to the Army's Suicide Prevention and Intervention Training: An Interagency Collaboration to Develop the New "ACE Base +1" Curriculum.

Suicide is a prevalent problem impacting the military community. The U.S. Army recognized the need to address this complex issue; one line of effort has been to provide suicide prevention and intervention education and training that is informed by current research, doctrine, and implementation best practices. The purpose of this article is to outline and present the genesis of the Army's new suicide prevention and intervention training-"Ask, Care, Escort (ACE) Base +1"-that aligns with the DoD newly published regulation-driven initiatives. The development of the "ACE Base +1" curriculum was a collaborative effort between two organizations within the Defense Health Agency: The Defense Centers for Public Health-Aberdeen and the WRAIR. A multidisciplinary team was formed by selecting specific operational and subject-matter experts from each organization based on educational qualifications and practical experiences expected to aid the development of curriculum content (e.g., subject-matter experts) and/or the training design. Revisions to the curriculum were informed by the Army's existing suicide prevention training module, current research in suicide prevention and public health, updated and relevant Army regulations, and current public health policy guidance from the CDC and the Department of Suicide Prevention Office. A detailed account of the systematic and iterative curriculum development process is provided. The interagency collaborative efforts resulted in a suite of training products, "ACE Base +1" version 1.3 that is modernized in training content, delivery methods, and design. Four primary elements shaped the final products: (1) A modular framework allowing a tailored approach to mandatory training, (2) a public-health approach that focuses on earlier intervention opportunities while building trust and cohesion, (3) a training design centered on peer discussions and behavioral rehearsal, and (4) an expansion of the curriculum to be inclusive of the entire Army community. Practical implications for each element are discussed.As the program of record, "ACE Base +1" training satisfies the annual requirement for all Active Army, Army National Guard, U.S. Army Reserve, and Department of the Army (DA) civilians. Both the training content (e.g., public-health concepts) and design of "ACE Base +1" reflect a comprehensive approach, focused on developing concrete, applicable skills that support the shared responsibility to suicide prevention and intervention. Limitations, such as delayed interagency collaboration and time constraints, are discussed. Future directions include recommendations for future curriculum projects, specifically within military populations, such as interprofessional, interagency collaboration, and selecting a multidisciplinary team of subject-matter experts. Additionally, WRAIR plans to continue their support to Directorate of Prevention, Resilience and Readiness with the expansion of the +1 menu of trainings, ongoing program evaluation, and longitudinal analysis to inform future revisions and ensure the content and delivery methods remain modernized, relevant, and effective.

Realist analysis of qualitative data in health professions education research.

Qualitative realist analysis is gaining in popularity in health professions education research (HPER) as part of theory-driven program evaluation. Although realist approaches such as syntheses and evaluations typically advocate mixed methods, qualitative data dominate currently. Various forms of qualitative analysis have been articulated in HPER, yet realist analysis has not. Although realist analysis is interpretive, it moves beyond description to explain generative causation employing retroductive theorising. Ultimately, it attempts to build and/or 'test' (confirm, refute or refine) theory about how, why, for whom, when and to what extent programs work using the context-mechanism-outcome configuration (CMOC) heuristic. This paper aims to help readers better critique, conduct and report qualitative realist analysis. We describe four fundamentals of qualitative realist analysis: (1) simultaneous data collection/analysis; (2) retroductive theorising; (3) configurational analysis (involving iterative phases of identifying CMOCs, synthesising CMOCs into demi-regularities and translating demi-regularities into program theory); and (4) realist analysis quality (relevance, rigour, richness). Next, we provide a critical analysis of realist analyses employed in 15 HPER outputs-three evaluations and 12 syntheses. Finally, drawing on our understandings of realist literature and our experiences of conducting qualitative realist analysis (both evaluations and syntheses), we articulate three common analysis challenges (coding, consolidation and mapping) and strategies to mitigate these challenges (teamwork, reflexivity and consultation, use of data analysis software and graphical representations of program theory). Based on our critical analysis of the literature and realist analysis experiences, we encourage researchers, peer reviewers and readers to better understand qualitative realist analysis fundamentals. Realist analysts should draw on relevant realist reporting standards and literature on realist analysis to improve the quality and reporting of realist analysis. Through better understanding the common challenges and mitigation strategies for realist analysis, we can collectively improve the quality of realist analysis in HPER.

Adoption of self-measured blood pressure monitoring in underserved communities: Program evaluation in primary care.

Self-measured blood pressure (SMBP) monitoring is crucial for hypertension management, yet its adoption, particularly among disadvantaged populations, remains low. 'Love Your Heart' is a quality improvement program providing free standard SMBP devices to hypertensive patients, aiming to enhance adoption and assess its impact on blood pressure control. This study evaluates the 'Love Your Heart' program's implementation success through assessing adoption rates and exploring changes in systolic blood pressure (SBP) among participants. We aim to understand factors influencing adoption and potential benefits of SMBP monitoring in a diverse, socially disadvantaged patient population. We retrospectively evaluated the 'Love Your Heart' program using electronic health records (EHR) at a primary care site in Chicago. Adult patients with hypertension were enroled in the 6-month program, which included education sessions and free SMBP devices. Adoption was measured by participation in program components, and changes in SBP were analysed based on adoption status. Statistical analyses were conducted using SAS software, adhering to STROBE reporting guidelines. Of 621 eligible patients, 104 participated, with 83 included in the evaluation. Despite all participants receiving free SMBP devices, adoption rates were modest, with only 7% sharing readings with the care team. However, patients who received device instructions demonstrated greater decreases in SBP compared to those who did not. Although not statistically significant, clinically meaningful decreases in SBP were observed among adopters. The 'Love Your Heart' program highlights the challenges of promoting SMBP monitoring among disadvantaged patient populations. While providing free devices addresses access barriers, low reporting to the care team suggests the need for further support mechanisms. Future research should explore strategies to enhance SMBP adoption and integration into clinical care, particularly in settings lacking automated data transmission systems.

Supporting care providers through a holistic telehealth intervention: a mixed methods evaluation of the Integrative Touch TeleWellness program

ABSTRACT To address the needs of frontline healthcare workers and family-based caregivers during the coronavirus pandemic, Integrative Touch developed a holistic TeleWellness Program consisting of a variety of mind-body techniques over 10 weekly sessions. This mixed methods study evaluates the feasibility, outcomes, and client experience of the TeleWellness Program during the first 2 years of the pandemic. The study found preliminary evidence of positive changes during the intervention and meta-inferences regarding care providers becoming better able to provide care by addressing their own well-being first. These findings have meaningful implications for future research on accessible, holistic interventions to support care providers.

A qualitative evaluation of a fentanyl patch safer supply program in Vancouver, Canada

BackgroundThe ongoing overdose crisis in Canada has prompted efforts to increase access to a “safer supply” of prescribed alternatives to the unregulated drug supply. While safer supply programs predominantly distribute hydromorphone tablets, the Safer Alternatives for Emergency Response (SAFER) program in Vancouver, Canada offers a range of prescribed alternatives, including fentanyl patches. However, little is known about the effectiveness of fentanyl patches as safer supply. Drawing on the perspectives and experiences of program participants, we sought to qualitatively evaluate the effectiveness of the SAFER fentanyl patch program in meeting its intended aims, including reducing risk of overdose by decreasing reliance on the unregulated drug supply. MethodsAs part of a larger mixed-methods evaluation of SAFER, semi-structured qualitative interviews were conducted with 17 fentanyl patch program participants between February 2022 and April 2023. Thematic analysis of interview data focused on program engagement, experiences, impacts, and challenges. ResultsThe flexible program structure, including lack of need for daily dispensation, the extended missed dose protocol, and community pharmacy patch distribution fostered engagement and enhanced autonomy. Improved management of withdrawal symptoms and cravings due to steady transdermal dosing led to reduced unregulated drug use and overdose risk. Participants also experienced economic benefits and improvements in overall health and quality of life. However, skin irritation and patch adhesion issues were key barriers to program retention. ConclusionOur findings demonstrate the value of including fentanyl patch safer supply in the substance use continuum of care and offer insights for innovations in delivery of this intervention.

Pemanfaatan Card Game untuk Meningkatkan Minat Belajar Bahasa Inggris Siswa-Siswi Kelas 8 di SMP Negeri 2 Kaliwungu

Fulfilling the English learning needs of junior high school students by using qualified teaching aids is not easy for English teachers in rural Indonesia. To overcome this obstacle, SMP Negeri 2 Kaliwungu Boyolali collaborated with a team of four students and one lecturer of PBI UKSW in the program "Visual Learning Aids for English Teaching in SMP Negeri 2 Kaliwungu." This community service program aims to help English teachers of SMP Negeri 2 Kaliwungu create effective and fun English learning by providing card game learning aids based on the needs analysis of the English teachers and students, especially Grade 8. This community service program generally consisted of four steps: preparation, interviewing, making teaching media, and evaluation, which included presenting the teaching media to the school and a game simulation with some students. The program evaluation results showed that the students could understand the story by arranging the sentences on the card game correctly, and the teacher was also satisfied with the alternative learning media because the attractive visuals on the cards could motivate students.

Evaluation Policy Study of the Second Round of the Taiwan Teacher Preparation Program Evaluation.

This study focuses on the second round of Taiwanese teacher preparation program (TPP) evaluation (2012-2017) to analyze how evaluation policy shapes TPPs. In-depth interviews were conducted with 13 stakeholders, which included professors, administrators, researchers, and staff involved in TPP evaluation. First, the findings revealed that neoliberalism influences evaluation policy and restricts the diversity of TPP development. Second, different evaluation approaches may be used to bring more inclusiveness and flexibility to evaluation standards. Third, evaluation plays an important role in TPP development, and a professional organization responsible for evaluation should be established in Taiwan in the future. To conclude, it is significant for TPP evaluation policy to consider policy ideology, evaluation standards, and program diversity to aid in improving the quality of teacher education not only in Taiwan but also internationally.

Evaluation of an ODL Contemporary Novel Program for Sudanese EFL University Students During the Wartime: A Case Study of Khartoum University

Evaluation of an ODL Contemporary Novel Program for Sudanese EFL University Students During the Wartime: A Case Study of Khartoum University

A Realistic Evaluation of Homicide Reduction Programmes

A Realistic Evaluation of Homicide Reduction Programmes

The Study of APIPA Model Concept Development through the Application of Analytical Hierarchy Process (AHP)

Improving the quality of teacher performance programs certainly requires the important role of teachers and school leaders. The role of teachers can be seen in the implementation of performance programs which require systematic and structured evaluation of performance programs. The results of the evaluation and monitoring of the implementation of the teacher performance program by school leaders are used as material for consideration in making decisions about the quality level of the teacher performance program. Making this decision is very difficult for a school leader in evaluating teacher performance programs. In this case, a modified program evaluation was developed by the CIPP model program evaluation with ALKIN model program evaluation. The research used Development Research (R&D) using 4D design (Define, Design, Development, Dissemination). The Define stage conducts a literature review regarding weaknesses in the CIPP and ALKIN model program evaluation syntax. The Design stage prepares the topology related to the main ideas of the program evaluation development syntax. The Development stage carries out operational definitions and indicators for each program evaluation syntax. The dissemination stage carries out an expert judgment or assessment of the experts who are analyzed using the analytical Hierarchical Process (AHP) to determine the consistency of the program evaluation development concept. The results using AHP are consistent with the concept of developing the APIPA model program evaluation which has the syntax Assessment, Planning, Implementation, Product, and Appreciation for evaluating teacher performance programs.

From Unemployment to Self-Employment: An Evaluation of Self-Employment Assistance Programs

From Unemployment to Self-Employment: An Evaluation of Self-Employment Assistance Programs

Double-Ended Bit-Stealing for Algebraic Data Types

Algebraic data types are central to the development and evaluation of most functional programs. It is therefore important for compilers to choose compact and efficient representations of such types, in particular to achieve good memory footprints for applications. Algebraic data types are most often represented using blocks of memory where the first word is used as a so-called tag, carrying information about the constructor, and the following words are used for carrying the constructor's arguments. As an optimisation, lists are usually represented more compactly, using a technique called bit-stealing, which, in its simplest form, uses the word-alignment property of pointers to byte-addressed allocated memory to discriminate between the nil constructor (often represented as 0x1) and the cons constructor (aligned pointer to allocated pair). Because the representation supports that all values can be held uniformly in one machine word, possibly pointing to blocks of memory, type erasure is upheld. However, on today's 64-bit architectures, memory addresses (pointers) are represented using only a subset of the 64 bits available in a machine word, which leave many bits unused. In this paper, we explore the use, not only of the least-significant bits of pointers, but also of the most-significant bits, for representing algebraic data types in a full ML compiler. It turns out that, with such a particular utilisation of otherwise unused bits, which we call double-ended bit-stealing, it is possible to choose unboxed representations for a large set of data types, while still not violating the principle of uniform data-representations. Examples include Patricia trees, union-find data structures, stream data types, internal language representations for types and expressions, and mutually recursive ASTs for full language definitions. The double-ended bit-stealing technique is implemented in the MLKit compiler and speedup ranges from 0 to 26 percent on benchmarks that are influenced by the technique. For MLKit, which uses abstract data types extensively, compilation speedups of around 9 percent are achieved for compiling MLton (another Standard ML compiler) and for compiling MLKit itself.

Benefits and Challenges of Addressing Cultural Intersectionality for Program Development and Evaluation.

Program development and evaluation that promote health equity are driven by community-based participatory research and led by people from select communities to adhere to the principles of diversity, equity, inclusion, and accessibility (DEIA). This process is enriched when these communities represent intersections within DEIA, such as, for example, Korean American women with depression. Research shows that culture-specific adaptations of services are significantly more effective for these communities than is the standard of care for the broader population. However, cultural specificity decreases generalizability of findings and scalability of interventions. Decisions about intersectionality are affected by implications for science as well as real-world benefits for individuals participating in this process.

Overcoming Poverty Through Social Programs: Evaluation of Effectiveness and Implementation

Poverty is a complex and multidimensional social issue that requires special attention in efforts to overcome it. This study aims to evaluate the effectiveness and implementation of various social programs designed to reduce poverty. This study focuses on the analysis of various social programs in several developing and developed countries, as well as assessing the extent to which these programs are successful in achieving their goals. The research method used is a qualitative approach with literature study and library research. Data sources consist of program evaluation reports, journal articles, books, and relevant policy documents. Through an in-depth analysis of these documents, this study identifies key factors that affect the effectiveness of social programs, including planning, implementation, and evaluation of outcomes. The results show that although many social programs have good intentions and have been implemented with adequate resources, there is significant variation in their effectiveness. Factors such as community engagement, adaptation to local contexts, and coordination between institutions have proven to play a crucial role in determining the success of the program. The study also found that consistent implementation and ongoing assessment are essential to ensure a positive and sustainable impact.