Program Evaluation Research Articles

Evaluation of the Power Up program: a health promotion program encouraging healthy lifestyle habits among youth in summer day camps—study protocol

Evaluation of the Power Up program: a health promotion program encouraging healthy lifestyle habits among youth in summer day camps—study protocol

Evaluation of the social inclusion programme in Slovenia

Evaluation of the social inclusion programme in Slovenia

Peak Team: Survey of Student Strengths and Needs and Teacher’s Evaluation of an Innovative Trauma-Informed School Program

Peak Team: Survey of Student Strengths and Needs and Teacher’s Evaluation of an Innovative Trauma-Informed School Program

Content Analysis of Social Determinants of Health Accelerator Plans Using Artificial Intelligence: A Use Case for Public Health Practitioners.

Public health practice involves the development of reports and plans, including funding progress reports, strategic plans, and community needs assessments. These documents are valuable data sources for program monitoring and evaluation. However, practitioners rarely have the bandwidth to thoroughly and rapidly review large amounts of primarily qualitative data to support real-time and continuous program improvement. Systematically examining and categorizing qualitative data through content analysis is labor-intensive. Large language models (LLMs), a type of generative artificial intelligence (AI) focused on language-based tasks, hold promise for expediting content analysis of public health documents, which, in turn, could facilitate continuous program improvement. To explore the feasibility and potential of using LLMs to expedite content analysis of real-world public health documents. The focus was on comparing semiautomated outputs from GPT-4o with human outputs for abstracting and synthesizing information from health improvement plans. Our study team conducted a content analysis of 4 publicly available community health improvement plans and compared the results with GPT-4o's performance on 20 data elements. We also assessed the resources required for both methods, including time spent on prompt engineering and error correction. Accuracy of data abstraction and time required. GPT-4o demonstrated abstraction accuracy of 79% (n=17 errors) compared to 94% accuracy by the study team for individual plans, with 8 instances of falsified data. Out of the 18 synthesis data elements, GPT-4o made 9 errors, demonstrating an accuracy of 50%. On average, GPT-4o abstraction required fewer hours than study team abstraction, but resource savings diminished when accounting for time for developing prompts and identifying/correcting errors. Public health professionals who explore the use of generative AI tools should approach the method with cautious curiosity and consider the potential tradeoffs between resource savings and data accuracy.

Retrospective Evaluation of Fluoride Varnish Application Programs and Their Impact on Early Childhood Caries

Retrospective Evaluation of Fluoride Varnish Application Programs and Their Impact on Early Childhood Caries

A avaliação de revistas jurídicas no Brasil: perspectivas futuras em um cenário “pós-Qualis”

The extinction of “Qualis Periódicos” by CAPES marks a transformation in the evaluation of scientific article production in Brazil, particularly in the field of Law. For years, Qualis was the main (and perhaps the only) criterion for classifying law journals in Brazil, influencing both scholars’ choice of periodicals and the evaluation of stricto sensu Graduate Programs. The system brought significant advancements, such as increased objectivity in evaluation, greater recognition of scientific journals, and the strengthening of peer review. However, it also faced criticism, including a lack of predictability in the early evaluation cycles, delays in classification, and the imposition of uniform criteria that were inadequate for certain fields. With CAPES' new methodology, the evaluation aims to focus on individual articles rather than just the journals that publish them. Each evaluation area will have the autonomy to adopt different criteria, combining bibliometric indicators, qualitative analysis of articles, and qualitative criteria for journals. In the case of Law, indexing in recognized databases such as Scopus, Web of Science, SciELO, and Redalyc emerges as a relevant parameter for assessing publication quality. The adaptation of law journals to the requirements of these indexing databases becomes essential to expanding the number of indexed periodicals. Measures such as improving editorial practices and adhering to international indexing standards are fundamental to ensuring the continued recognition of law journals within CAPES' new evaluation framework.

Evaluation of PT Petrokimia Gresik's LONTAR Program With SROI (Social Return On Investment)

This study evaluates the LONTAR Program (Kelompok Jajanan Masyarakat Sekitar) by PT Petrokimia Gresik in 2023 using the Social Return On Investment (SROI) method. This program aims to empower MSMEs through skills training and free bazaar facilities. SROI analysis is carried out using the Present Value, Future Value, and Real Value approaches to measure social, economic, and environmental impacts. The results show SROI values of 2.20 (Present Value), 2.07 (Future Value), and 2.08 (Real Value), which means that every investment of IDR 1.00 generates a social return of IDR 2.20, IDR 2.07, and IDR 2.08. These findings indicate a positive and sustainable impact, especially in increasing the income and skills of MSMEs. The study recommends scaling up programs and ongoing training to maximize their impact, while encouraging the adoption of SROI methods in the evaluation of corporate social responsibility programs.

Evaluasi Program Penerimaan Mahasiswa Baru di Instansi Pendidikan dengan Pendekatan Sosialisasi

This study aims to determine the evaluation of the new student admission program with a socialization approach. Socialization is a way to promote the organization directly. The goal is to describe socialization in new student admissions in educational institutions that can explain the context, input, process, and products that exist in the university environment. So that it can see the evaluation of the public relations program carried out. This research is a literature study to understand and study theories from various literatures relevant to the research. This literature technique uses content analysis as a data analysis technique. The evaluation carried out looks at the socialization that describes the background through the context, the evaluation that assesses the existing indicators through inputs, resources, facilities and infrastructure to achieve program goals, the evaluation process is carried out by SNMPTN, SBMPTN, and Independent pathways, while product evaluation is carried out by assessing the suitability of campus capacity.

Evaluation of the Slovenian Breast Cancer Screening Programme: Years of Life Gained and Avoided Deaths.

Background: Most commonly, mortality is used for evaluation of organized breast cancer screening programmes. Survival analyses are also being employed to provide more timely assessment. A novel approach has been introduced for calculating survival using Slovenian data. Methods: Breast cancer cases from the Slovenian Cancer Registry between 2008-2021 in women aged 50 to 72 were included, supplemented by data on the screening history from the screening registry. The comparison was made in two scenarios-invited or not invited to the screening. Survival, years of life gained, and number of avoided deaths over 10 years were calculated and standardized. The comparison was adjusted for confounding covariates, and a biological tumour growth model was used to adjust the survival of the invited group for lead time bias and over-detection. Results: A total of 9392 breast cancer cases were included in the study. Estimated median lead time for screen-detected cases was 1.23 years. The survival probability accounting for lead time bias and covariates of cancers patients invited to the programme was 4.3 percentage points higher than the not invited (0.81 vs. 0.76). Standardized to 100,000 women in the 2-year period, this would result in 22 avoided deaths. On average, the invited cancer patient lived 0.22 years longer, which amounts to a total of 114 years of life gained. Conclusions: The results show the overall benefit of the breast cancer screening programme. The standardized results enable cross-country comparisons. The calculation of years of life gained and avoided deaths can provide additional opportunities for communicating the results to non-expert populations.

Development and evaluation of an online training program for palliative care in India

BackgroundPalliative care training at the undergraduate level is poor in India. With the need for palliative care rising in India and globally, it is possible to train physicians in resource-limited settings in palliative care via online training programs owing to ease of access and convenience. However, there is limited evidence available from India on the development and implementation of such a training program. This study aims to describe the development of an online training program offered by the Trivandrum Institute of Palliative Sciences (TIPS), Kerala, and the impact of the program on the confidence of physicians in managing various palliative care needs of their patients.MethodsThe course was developed by an interdisciplinary expert team from TIPS. The course content was made keeping in mind the local sociocultural factors in India and was peer-reviewed by two external experts. The program was started in 2018 and updated and revised over the years. Currently, the program has 20 sessions, each lasting for 90 min. The course content was disseminated using project-ECHO’s (Extension of Community Health Outcomes) telementoring model. To assess the impact of the training program, self-reported change in confidence from twenty-two batches of physicians, trained from January 2020 to August 2023. Feedback from participants was also assessed to identify areas of improvement in the training program.ResultsA total of 1159 physicians were trained during the study period. At the end of the course, 51.2% of the participants completed the evaluation survey and had a statistically significant (p < 0.05) improvement in confidence in pain, gastrointestinal symptoms and breathlessness management, morphine prescription, and psychosocial communication. The duration of each session and the entire course was reported to be ideal by 88.6% and 87.9% of participants, respectively.ConclusionOur results show that online training can be effectively used to build confidence in physicians in managing various palliative care needs.Trial registrationNot applicable.

Evolution of Internal Quality Assurance Models for University Online Programs

The widespread adoption of online formats in 2020 established a trend towards the development of online education and intensified the issue of assessing the quality of distance higher education programs. The subsequent necessity to adapt to the post-pandemic context prompted universities to develop both new online programs and innovative approaches to managing their quality. Despite the proliferation of online formats, there is a lack of publications analyzing the evolution of internal quality assurance mechanisms in such contexts. This study examines the dynamics of the emergence and transformation of internal evaluation and quality assurance models for university online programs from 2009 to 2023. The research compares 17 models focused on various aspects of the educational process, thereby tracing the evolution of internal quality assurance mechanisms. The analysis is conducted based on parameters identified through a literature review as well as during the course of the research itself. The results allow for the identification of three developmental periods for these models: experimental, transitional, and contemporary. Educational aspects that are present in all models within the sample (technologies, student-centered design, and online teaching) are distinguished from those characteristic only of some models (environment development, resource reuse, support for students, faculty, and staff, and communication). A description of the evolution of each aspect within the identified periods is provided, along with their connection to the concept of quality in online environments. This work may be beneficial for researchers in the field of quality assurance, higher education administrators, and educational policy specialists.

China’s Experience in Accreditation as Lessons for Russia

This article is dedicated to the analysis of the quality assessment system for higher education in China, referred to as the “Five in One” framework. It comprises five interrelated mechanisms: institutional quality assessment, program accreditation, international accreditation of universities and programs, self-evaluation, and monitoring of the education system. The study emphasizes the significance of examining the specifics and developmental trends of the mechanisms and procedures for quality assessment in higher education in the People’s Republic of China as a condition for enhancing the preparation of highly qualified professional personnel. It is highlighted that the formation of a national quality assessment system for higher education is based on the adoption of best international practices while preserving and cultivating “Chinese specificity.” By comparing state educational policies regarding quality assessment, content, and procedures in China and Russia, the author concludes that there are divergent developmental trajectories. Furthermore, considering the necessity of strengthening cooperation between the two countries across all socio-economic spheres, including higher education, the analysis of differences in quality assessment approaches may be beneficial for further fostering interaction at both national and inter-institutional levels. The findings of this research hold particular significance in light of expanding collaboration and enhancing the mobility of students and faculty based on transparent mechanisms for recognizing educational quality, such as external evaluation and accreditation of educational programs.

What is the syllabus for? Revealing tensions through a scoping review of syllabus uses

ABSTRACT The syllabus, also known as the course plan, is a document rooted in the interaction between instructors and learners in the classroom, yet it serves additional functions within the institution. Its expanding application beyond the classroom to address accountability and program evaluation underscores the importance of acknowledging the multifaceted nature of the syllabus, housing diverse requirements and goals that may not always seamlessly align. For instance, the imperative for a succinct and easily comprehensible document for learners may conflict with the document’s contractual function, necessitating specific length and tone considerations. Beyond examining its content, a comprehensive study of the collective uses of this tool offers insights into practices at the classroom, departmental, and institutional levels. This paper introduces a framework based on a scoping review of syllabus use in higher education literature. Employing an Activity Theory lens, nine interconnected uses are identified and categorized into three primary purposes. Analysis of these uses also sheds light on potential tensions related to syllabus use as a contract, blurred authorship, and the diverse stakeholders as audiences. By offering a broader perspective on the diverse functions of the syllabus, the presented framework can help inform discussions aimed at harmonizing the complex practices shaped by this institutional tool.

Readiness for scale up following effectiveness-implementation trial: results of scalability assessment of the Community Partnership Program for diabetes self-management for older adults with multiple chronic conditions

BackgroundImplementation research should assess the feasibility of scale up to bridge the evidence-practice gap for integrated care programs in the prevention and management of chronic conditions. Scalability assessment is the first critical step of scale up to determine the potential suitability of a promising health program to be adopted into routine practice and the fit of the program within local contexts. The Community Partnership Program (CPP), an integrated care intervention for older adults with diabetes and multiple chronic conditions, was designed at the outset with scale up in mind, and evaluated in an implementation-effectiveness randomized controlled trial across three Canadian provinces. The final phase of this program of research was to assess scalability and determine the critical factors and next steps for the development of a scale up plan.MethodsMultiple methods were used to assess the scalability of the CPP including collection and analysis of publicly available documents, synthesis of qualitative and quantitative evidence from studies of the CPP, semi-structured interviews with key informants, feedback and recommendations arising from working group meetings and knowledge exchange workshops to discuss and rate the scalability of the program. Data collection and analysis was informed by the Intervention Scalability Assessment Tool (ISAT); developed to support practitioners and policy makers in conducting systematic assessments of the suitability of health interventions for population scale-up in high-income countries.ResultsOverall, the CPP received high scalability ratings from participants. A phased, horizontal implementation and scale up process was recommended, facilitating local adaptations, on-going program evaluation, and accumulation of evidence. Challenges to scale up were identified, including the need for further evidence of program effectiveness in other diverse settings and populations, and designated funding and adequate health human resources.ConclusionsParticipants agreed the CPP meets the needs of many older adults with diabetes and multiple chronic conditions; however, they suggested further tailoring of the program to support different ethnocultural groups and targeting the CPP to older adults with higher needs. The scalability assessment process was a practical method to generate concrete strategies to facilitate the uptake of the CPP into practice.Trial registrationClinical Trials.gov Identifier NCT03664583. Registration date: September 10, 2018.

Monitoring vaccination coverage in Germany-current status and trends

Timely and reliable data on the vaccination status of the population are essential for the evaluation of vaccination programs. The analysis of school entry examination data and health claims data from the Associations of Statutory Health Insurance Physicians is the basis for monitoring the vaccination coverage in Germany. These two established and effective systems are supplemented by surveys among special target groups that can provide data on vaccination coverage, acceptance, and intention to be vaccinated.With the data from school entry examinations and health claims, vaccination coverage can only be reported with adelay of 1-2years. Furthermore, the health claims data are incomplete due to the increasing diversification of vaccination service providers. The need for acomplete and timely monitoring of the vaccination coverage during the COVID-19 pandemic revealed the limitations of these systems in an emergency situation. Therefore, the temporary Digital Immunization Monitoring (DIM) system for collecting COVID-19 vaccination data was established. With DIM, it was possible to collect COVID-19 vaccination data from all vaccination service providers on adaily basis.With the vision of timely and complete monitoring of vaccination coverage, the integration of vaccination data collection into the German Electronic Reporting and Information System for Infection Protection (DEMIS) is being developed as auniform reporting infrastructure. In the future, DEMIS will be used to collect timely, pseudonymized, and individual-based vaccination data from all vaccination service providers. This will enable the evaluation of vaccination programs with little delay using complete and high-resolution data as well as the rapid adaptation of vaccination recommendations and communication strategies as needed.

School Digitalization: Evaluation Program in Sekolah Penggerak at Samarinda City

Digitalization in education seeks to enhance the effectiveness of dynamic education, aligning with one of the driving school interventions: school digitalization. The sekolah penggerak program, implemented for four years, is a government effort to improve the quality of education in Indonesia. Since the implementation of school digitalization, the learning quality data in the sekolah penggerak education report has been labeled “medium” with an average achievement score of 58.02. Furthermore, there has been a general decline in the quality of learning since 2023 in elementary school units, as evidenced by the Samarinda City education report card for 2024. Therefore, it is imperative to evaluate the school digitalization intervention program. The research methodology employed is program evaluation using the CIPP model. The population of this study consisted of the 10 first-generation sekolah penggerak program. The willing participants in this research were 7 schools, with informants comprising 7 school principals, 7 lower-class teachers, and 14 upper-class teachers. The research findings revealed that the program was largely aligned with the government’s targets. However, several obstacles were identified. Notably, the provision of facilities and infrastructure was incomplete, and some educators lacked the necessary competency in digital education.

A qualitative evaluation of the reflective fostering programme - examining foster and kinship carers' experiences, practical application, and perceived impact.

Mentalization is the ability to think about and interpret behaviours of both self and others in terms of thoughts and feelings. Caregiver's capacity to mentalize can enhance the quality of parent-child relationships. The UK Reflective Fostering Programme (RFP) has been developed to enhance foster and kinship carers' self-regulation and self-mentalizing. To understand carers' experiences, practical application, and perceived impact of the RFP. A qualitative evaluation of the RFP using in depth interviews with twenty-four carers. Interviews were audio-recorded, transcribed verbatim and analysed thematically. Three key themes were identified: (1) 'Me time' - carer's reflections on the programme and what makes it work, highlights sharing of challenging experiences as important for practising mentalizing. (2) 'Stopping and thinking in the moment' - carer's understanding of programme concepts explores carer's conceptualizations of mentalization, revealing some male carers described challenges regarding sharing personal experiences in a group and the practical application of mentalization. (3) Practical application and perceived impact of the programme suggests successful enactment of learning from the programme, resulting in enhanced capacity to cope with stress, prevention of outbursts and better communication with children in their care. Our findings suggest the programme can successfully facilitate carer's use of self-mentalizing, leading to greater self-regulation and capacity to support children in their care. Future research should explore experiences of male and kinship carers to inform tailoring.

Imagining Futures: Evaluation of a blended programme of dialectical behaviour therapy and the creative arts for young women with a history of self-harm.

To evaluate an arts-enhanced dialectical behavioural therapy skills group for managing emotions and self-harm, implemented via an innovative public sector/third sector partnership to increase access to care. This is a pilot mixed-methods study. To assess participant experience in the 'Imagining Futures' programme, we examined self-report questionnaires and qualitative focus group interviews using framework analysis. We collected recruitment, session attendance and programme completion rates. To explore impact, we also report on quantitative psychological outcome measures, including self-harm frequency and overall well-being. We recruited 45 young women (mean age: 15.9, s.d. = 1.24, range 13.9-18.0 years) with a history of emotional dysregulation, self-harm and other contextual risks who were not receiving support from statutory child and adolescent mental health services in the United Kingdom. Participants were 22% not in education, employment or training and 77% were from United Kingdom racially minoritized backgrounds. The overall completion rate was 62% (n = 28/45). Qualitative data analysed from respondents (n = 25/28 young people and n = 12 parents) suggested high levels of satisfaction with the project. Thirteen themes were identified which described service elements perceived to support engagement and observed impact. There was an important role for relationships and the creative components. Quantitative clinical data indicated reductions in the frequency of self-harming, significant reductions in the perceived impact of difficulties and increased perceived social support. This novel delivery of a DBT skills group, incorporating blended psychology arts activities, has the potential to support engagement with psychological supports that improve mental well-being.

Pemberdayaan Kader Posyandu melalui Inovasi Pentol Lele Kreasi (Pentole Kres) untuk Penanggulangan Stunting di Kelurahan Tobimeita Kota Kendari

This community service program aims to improve the understanding and skills of Posyandu cadres in optimizing the use of catfish as a source of nutrition through the innovation of the Pentole Kres product to overcome stunting in Tobimeita Village, Kendari City. The method of implementing this service uses community empowerment through education and training activities. The program evaluation uses pre-test and post-test instruments which are analyzed using a paired t-test. The results of the service showed an increase in participant understanding of 85%, while 100% of participants experienced an increase in practical skills, and the average skill score increased from 2.4 before training to 4.5 after training. Based on the results of this community service program, it can be concluded that the community empowerment approach utilizing local potential, such as catfish, is effective in improving the understanding and skills of Posyandu cadres in managing nutritional sources to combat stunting.

Creating a Standardized Approach to Program Evaluation in a Magnet Recognized Pediatric Hospital System.

A standardized process for systematic program evaluation adds value to an organization. Using a goal-based framework, a standardized evaluation template was designed and piloted in the ambulatory setting of a 5-time Magnet®designated, regional, quaternary pediatric hospital system. Executive level decision-makers reported the template met implementation criteria for acceptability, appropriateness, and feasibility for use within the organization. Application of the template in a pilot program yielded valuable, cost-effective, unbiased formative and summative information for decision making.