Program Evaluation System Research Articles

Program evaluation of a specialized treatment home for adults with severe challenging behavior

AbstractIndividuals with intellectual and developmental disabilities who engage in severe challenging behavior may comprise 5%–10% of the clinical population. Unfortunately, challenging behavior literature tends to underrepresent adult participants and emphasize efficacy (Does the intervention work?) more often than effectiveness (Does the intervention work in real world settings?). We conducted a systematic program evaluation to examine the effectiveness of a comprehensive behavioral treatment package using a hybrid quasi‐experimental consecutive case series design featuring eight adults who experienced the treatment package. The results depicted a substantial decrease in challenging behavior and increase in adaptive behavior across sessions for most participants. Impressive treatment integrity outcomes (M = 84%, range, 82%–90%) showcased that the intervention was implemented as intended. Social validity surveys administered to participants, caregivers, and case managers provide support for the acceptability of the treatment package. Project limitations, clinical considerations, and future directions are discussed.

Learning from educators: Implementation of a positive behavior support program targeting challenging behavior in children with autism

Challenging behaviors (CB) are a frequent co-occurring problem in children with autism spectrum disorder (ASD) and hinder their response to recommended interventions such as early intensive behavioral intervention (EIBI). The Prevent-Teach-Reinforce for young children (PTR-YC) program was implemented to meet community-based EIBI educators’ training and support needs in managing CB in their day-to-day work with families. Although this positive behavior support program has a strong empirical basis, its implementation by community-based educators has yet to be assessed from a systematic and structured program evaluation perspective. Using Chen’s (2015) theoretical framework for program evaluation, this study assessed the quality of implementation of PTR-YC as perceived by 17 educators who received training and supervision on applying PTR-YC among families of children with ASD receiving EIBI services. Educators’ post-intervention interviews and questionnaires were analyzed using the logical model for program evaluation to identify obstacles and facilitators to the implementation of PTR-YC.

CDC-Funded HIV Testing Services Outcomes in Ending the HIV Epidemic in the U.S. (EHE) and Non-EHE Jurisdictions, 2021.

Ending the HIV Epidemic in the U.S. (EHE) aims to end the HIV epidemic by focusing on 57 jurisdictions most impacted by HIV. Using 2021 data from the National HIV Prevention Program Monitoring and Evaluation system, we calculated distributions of CDC-funded HIV tests and HIV testing services outcomes in EHE and non-EHE jurisdictions. We conducted chi-square tests and robust Poisson regression to compare differences in outcomes. Of the 1,753,873 tests conducted, a greater proportion were conducted in EHE (65.7%) versus non-EHE (34.3%) jurisdictions (p<0.001). Greater number of persons newly diagnosed were identified in EHE (n=5,861) versus non-EHE (n=2,329) jurisdictions; newly diagnosed positivity was higher in EHE (0.5%) versus non-EHE (0.4%; PR=1.31, 95% CI: 1.25-1.38) jurisdictions. Among persons newly diagnosed, there were no differences in proportion of linkage to care within 30 days in EHE jurisdictions (77.9%) versus non-EHE jurisdictions (77.1%; PR=1.01, 95% CI: 0.95-1.07). Proportion of persons newly diagnosed interviewed for partner services was lower in EHE (69.0%) versus non-EHE (84.8%; PR=0.81, 95% CI: 0.76-0.87) jurisdictions. CDC-funded HIV testing services in EHE jurisdictions conducted more HIV testing and diagnosed more persons. Jurisdictions can further expand HIV testing and related services to continue striving toward EHE goals.

Facilitating well-being: an examination of an online trauma-informed faculty/staff training designed to support college student resilience

College students report high psychological distress, mental health problems and anxiety. Research indicates that faculty and staff can help support these students in crisis, but there is limited trauma-informed training for higher education professionals who are not mental health counselors. This study explored an evidence-based online training program designed to help faculty/staff members facilitate well-being and resilience among college students. To test the training, a systematic program evaluation was conducted. Prior to completing the training, participants (n = 52) took a pretest. Then, after completing the training, they took a posttest. Results indicated a significant difference between pre and posttest. There were more positive attitudes toward trauma-informed approaches and resilience resources; stronger beliefs about prevalence and effects of psychological trauma; greater self-efficacy to perform trauma-informed approaches; and greater confidence in trauma-informed knowledge. Additionally, participants reported they were likely to use tools learned during the training. Thus, indicating that a self-paced, convenient, and culturally sensitive training may be beneficial for faculty and staff as they aid in student success. Practical implications, participant comments and specific suggestions for colleges are also discussed.

Motives Behind University Students’ Enrolment in the Co-operative Programs in Tanzania

This study explored cooperative education's dynamics at the Moshi Co-operative University in Tanzania, aiming to establish factors influencing students' enrolment in cooperative programs amidst challenges faced by Tanzanian cooperative societies. Utilizing Dweck's Goal Orientation Approach, the study employed a qualitative single-case design, involving 50 students selected through purposive sampling to ensure a diverse representation of perspectives and experiences. Data collection methods include semi-structured interviews and documentary reviews, facilitating insights into participants' motivations, experiences, and the contextual factors guiding their decisions. Thematic analysis guided by Dweck's approach ensured a systematic examination of the data, revealing the influential roles of parents, friends, teachers, family and role models in students' program choices. Emphasis is appears on the significance of employment opportunities and program relevance in shaping students' decisions. The study concludes that influential factors in students' enrolment include parental, peer, teacher, family and role model factors, emphasizing the significance of employment opportunities and program relevance. To strengthen cooperative education at MoCU, the study recommends collaborative initiatives, program diversification, improved career guidance, continuous professional development, systematic program evaluation, policy refinement and longitudinal impact assessments.

CDC-Funded HIV Testing and Undiagnosed HIV Infection in Ending the HIV Epidemic in the U.S. Jurisdictions.

Approximately 13% of persons with HIV (PWH) are unaware of their status. To help end HIV, it is important to understand the relationship between the rates of HIV testing and undiagnosed HIV infection. Ending the HIV in the U.S. (EHE) jurisdictions. Using 2021 data from the National HIV Surveillance System and the National HIV Prevention Program Monitoring and Evaluation system, we calculated estimated rates of undiagnosed HIV infections per 100,000 population and rates of CDC-funded HIV tests per 1,000 population. We assessed the association between the two rates using Spearman's rank correlation. We also calculated a rank difference between the two rates for each EHE jurisdiction to help identify jurisdictions with greater unfilled needs for HIV testing. Overall, CDC-funded HIV tests per 1,000 population were positively associated with estimated rate of undiagnosed HIV infection per 100,000 population (rho=0.55, p<0.001). EHE jurisdictions with the greatest magnitude of negative rank differences (i.e., higher undiagnosed HIV infection per 100,000 population and lower CDC-funded HIV tests per 1,000 population) were Prince George's County, MD; Mecklenburg County, NC; Hudson County, NJ; Bronx County, NY, and Hamilton County, OH. In general, CDC-funded HIV testing is being conducted in jurisdictions with the greatest needs. However, we also found large discrepancies between CDC-funded HIV testing and undiagnosed HIV infection rates for some jurisdictions. These jurisdictions may want to identify barriers to their HIV testing services and expand their programs to ensure that all PWH in their jurisdictions are diagnosed.

‘First, do no harm’: systematic program evaluation of an equine veterinary service-learning initiative with Indigenous communities in Canada

BackgroundVeterinary students have historically lacked meaningful experiential learning opportunities in equine medicine. At the same time, there are barriers to accessing veterinary care in Indigenous communities stemming from colonial injustices. In 2018–2019, a partnership was initiated where University of Calgary students began to provide equine veterinary services to Indigenous communities. As the first-documented equine veterinary service-learning initiative in Indigenous communities embedded in a veterinary curriculum, the purpose of the study is to systematically evaluate the program for its potential impact as part of a formative process for improvement.MethodsMultiple parties in the program were engaged in a convergent, parallel, mixed-methods systematic program evaluation to explore the main program outcomes: (1) equine veterinary care; (2) clinical experiential student education; (3) cultural training of veterinary professionals and students; and (4) education of community members. The hypothesis was that ethical development using the “first, do no harm principle” would lead to benefits including a healthy horse population, a technically and culturally competent veterinary community, and an educated horse clientele.ResultsThe program had a positive impact on accessibility to veterinary care and self-reported improvement in veterinary and cultural competency. In addition to the hypothesized program outcomes, additional program outcomes and effects were identified, including reciprocal learning and relationship building with the Indigenous community, leading to trust and equity-building. The students learned from both the in-community programming as well as the Indigenous community members they worked with.ConclusionProgram evaluation of an equine service-learning initiative in Indigenous communities reveals multiple and profound impacts including improved patient health status, wider scope of veterinary and cultural learning, strengthened relationships, and reciprocal learning with partnering Indigenous communities.

Budget Program Evaluation in Armenia as a Tool to Improve the Effectiveness of Public Spending

In the last decade, like in many countries of the world, as well as in Armenia, significant budget reforms have taken place. An important part of those reforms is the introduction of the budget program evaluation system. The paper discusses the problems of the Armenian budget system and importance of introduction of evaluation system, as well as reviews alternative models of the introduction of this system. Thus, the purpose of the article is to develop theoretical, methodological, as well as practical recommendations regarding the implementation of the budget program evaluation system in RA as a tool to improve the efficiency of public spending. By examining successful experience, the article seeks to offer actionable insights for policymakers, government officials, and stakeholders, thus the article highlight the importance of stable and transparent budget program evaluation system in ensuring accountability, resource optimization, and the delivery of tangible benefits to the Armenian population. The article also addresses challenges in the introduction of an evaluation system, such as data limitations, institutional capacity, and stakeholder engagement.

Unlocking the transformative power of mentoring for youth development in communities, schools, and talent domains.

Mentoring is a highly individualized educational measure that can support youth development in communities, schools, and talent domains. Depending on the target population, goals, structure, and medium, mentoring for youths can differ considerably. This article first reviews the main types of mentoring programs and practices for youth development in communities, schools, and talent domains. Despite the popularity of mentoring programs, many programs fail to realize the full potential of mentoring as meta-analyses consistently show relatively small effects of mentoring. The discrepancy between the potential and actual effect of mentoring is referred to as the mentoring paradox. Crucial aspects that are held responsible for the mentoring paradox, such as adequate planning and implementation of mentoring programs, adherence to research-based mentoring practices, as well as quality assurance of mentoring programs through systematic program research and evaluation are described. Finally, implications on how to professionalize mentoring are provided for different stakeholders.

Peculiarities of evaluating the results of implementation of regional development programs in Ukraine

Purpose. To systematize current problems in the field of using modern methods for evaluating the effectiveness of regional development programs in Ukraine and to substantiate the scientific foundations of program evaluation methodologies in accordance with modern principles of program-target planning. Methodology. In the process of work, generally accepted methods of scientific economic research were used, namely, systematic and comparative analysis of scientific literature, practices of development and implementation of spatial development programs in Ukraine and abroad, generalization, methods of logical abstraction and generation of conclusions. Findings. As a result of the conducted research, a problem is established of the lack of a regional development program evaluation system in Ukraine, which reduces the effectiveness of the creation and implementation of such programs. The research lays out the foundations of a scientifically based methodology for monitoring and evaluating regional development programs, which has been tested in economically developed countries and requires further scientific elaboration and introduction into the practice of state regulation of post-war reconstruction and socio-economic development of the regions of Ukraine. Originality. The scientific and practical prerequisites and the current state of the problem of forming a modern system for assessing the effectiveness of regional development programs in Ukraine are revealed. The concept of the theory and methodology of forming a modern evaluation mechanism of post-war reconstruction and development programs of Ukraine’s regions is put forward, theoretically justified and confirmed by the practice of regional management in economically developed countries. Practical value. The introduction of the proposed methodology for the development and implementation of regional development programs will ensure compliance of the practice of regional management in Ukraine with EU standards, which will contribute to European integration processes; will increase the capabilities of state authorities and local governments to attract funds for post-war reconstruction programs and further development of Ukraine’s regions; will ensure transparency of the use of funds in the process of program implementation and reduce the level of corruption risks; will increase the efficiency of program implementation in general.

Investigating the impacts of a preservice agriculture teacher recruitment program using Kirkpatrick’s program evaluation model

Although post-secondary school agricultural education degree programs strive to meet the need of supplying agriculture teachers, the gap between teacher demand and supply continues to exist. The Teach Ag Campaign is a joint effort of the agricultural education profession to recruit and retain agricultural educators and encourage students to pursue a career in teaching agriculture. Given the systematic program evaluation of the effects of Teach Ag Campaign efforts on agriculture teacher supply is lacking, this study investigated the outcomes and effectiveness of a preservice agriculture teacher recruitment program using Kirkpatrick’s program evaluation framework. The results indicated that the design and implementation of the Agricultural Education Institute (AEI) effectively met the needs of program participants, and the program positively affected participants’ knowledge, attitude, and aspiration regarding a career teaching agriculture. In addition, the findings demonstrated that the AEI contributes to producing agriculture teachers who support Florida's school-based agricultural education programs. Based on the results of this study, recommendations for practice in terms of participant recruitment, program design, implementation, and future research were proposed.

Evaluation Model Of Noble Moral Education For Students In Madrasah

This article aims to find out the model for evaluating noble character education by schools or institutions in instilling noble character values for students in the madrasah environment. The background of this research is that teachers and institutions have not carried out a systematic evaluation of the noble character education program for students. This type of research is program evaluation research. This evaluation is used to evaluate the noble character education program for students at Madrasah Aliyah Negeri in Bima. The approach used in this type of research is a quantitative and qualitative descriptive approach. The subjects in this evaluation were class XII students, the head of the Madrasah, Class XII Aqidah Akhlak Subject Teachers, and the Madrasah Committee at Madrasah Aliyah Negeri Bima. The data sources used in this study were questionnaires, documentation and interviews. The results of this study indicate that the evaluation model that is suitable for use by teachers and institutions is the Stake's Countenance evaluation model developed by Robert Stake. The Stake's countenance evaluation model consists of three stages, namely the antecedent (preparation stage), transaction (implementation stage), and outcome (result) stage. And this model is very appropriate for evaluating the noble character education program at Madrasah Aliyah in Bima, because these stages are interrelated with one another.

Towards the Establishment of Program Evaluation System in Japanese Higher Education: Learning from the Quality Code Adopted in the United Kingdom

Towards the Establishment of Program Evaluation System in Japanese Higher Education: Learning from the Quality Code Adopted in the United Kingdom

Developing evidence-based resources for evaluating postgraduate trainees in the biomedical sciences.

Postgraduate trainees elevate the academic strength of institutions by conducting research, promoting innovation, securing grant funding, training undergraduate students, and building alliances. Rigorous and systematic program evaluation can help ensure that postgraduate training programs are achieving the program's intended outcomes. The purpose of this project was to develop evidence-based evaluation tools that could be shared across federally funded biomedical training programs to enhance program evaluation capacity. This manuscript describes the evidence-based process used to determine program evaluation needs of these programs at a research-intensive university. Using a multi-phased sequential exploratory mixed methods approach, data were collected from trainees, employers, leaders, and program directors. Data analyses included document analysis of program plans, inductive coding of focus groups and interviews, and descriptive analysis of surveys. Two overarching categories-Trainee Skills and Program Characteristics-were identified including six themes each. Program directors prioritized communication, social and behavioral skills, and collaboration as the trainee skills that they needed the most help evaluating. Furthermore, program directors prioritized the following program characteristics as those that they needed the most help evaluating: training environment, trainee outcomes, and opportunities offered. Surveys, interview scripts, and related resources for the categories and themes were developed and curated on a publicly available website for program directors to use in their program evaluations.

EFL Students’ Online Course Satisfaction and Attitudes Towards E-Learning in Post-Pandemic Hybrid Education

Purpose: This study seeks to identify the EFL students’ satisfaction with the online courses they took within the scope of hybrid education in the post-pandemic period and their attitudes towards e-learning, and to determine whether there is a relationship between these two variables. &#x0D; Design/Methodology/Approach: Students studying in Iğdır University English Language and Literature Undergraduate Program in the spring semester of the 2021-2022 academic year constitute the population of this research. Since there are online courses at all levels, including the preparatory program in the department, all of the department students were included in the research and in order to collect the research data, the Online Course Satisfaction Scale and the Attitudes towards E-learning Scale were administered at four levels (Preparation, 1st, 2nd, and 3. Grades) of the program. The data obtained were transferred to the SPSS program and descriptive and inferential analyzes were carried out.&#x0D; Findings: The research findings indicated that the students were moderately satisfied with their experiences in online courses and they had mildly positive attitudes towards e-learning. The students’ satisfaction was not affected by their gender, nor their perceived L2 proficiency, but by the year of study, juniors and preparatory year students having the highest satisfaction means. As regards the learner attitudes, male students had significantly more positive attitudes towards e-learning than females and the sophomores and juniors reported more positive attitudes than students at other levels. A statistically significant relationship between online course satisfaction and attitudes towards e-learning was identified as a result of the analyses. &#x0D; Highlights: The moderate satisfaction and attitude levels found as a result of the study suggest that the challenges the students encountered during the emergency remote teaching might be continuing in hybrid education practices. The implementation of more solid, realistic and systematic program evaluation is suggested in order to mitigate the factors adversely affecting EFL students’ satisfaction and attitudes towards e-learning.

A Standards-Based Checklist for Program Evaluation as Scholarly Activity.

Educators in graduate medical education (GME) regularly make decisions about adding, deleting, or revising some aspect of a curriculum, rotation, or other program activity. At the core of each decision is the activity's value to the learners, faculty, program, and sponsoring institution, and/or to meeting accreditation/licensure requirements. When these decisions are informed by systematic program evaluation, the results may be generalizable and have value to others in GME beyond their local worth. Yet how does an educator know if their work might be relevant to others or fulfill scholarship criteria? These are particularly vexing challenges for evaluation, as every program is local and has unique settings, trainees, faculty, and resources. Thus, disseminating evaluation as scholarship is a balancing act: staying true to the local nature of the program while addressing concerns of skeptics who may say, “That's not relevant…it wouldn't work here.”Applying standards of program evaluation can tip the balance toward becoming a scholarly activity. Other Rip Outs1,2 in this series outlined the American Evaluation Association (AEA) standards (accuracy, feasibility, integrity [ie, propriety], and utility)3 and suggested their application to program planning and implementation. Viewing these same standards in a checklist format can guide evaluation as a scholarly activity for authors and reviewers. Reminder: standards apply to evaluation, not to the project (eg, asking if the evaluation is feasible, not if the project is feasible). Standards and sample items include:As evaluators, readers, and editors, we like checklists ordered to match the standard introduction/aim, methods, results, discussion/conclusions (IMRD) framework. However, evaluation standards underpin the entire program evaluation endeavor; thus, one standard can—and often should—appear in multiple sections. See online supplementary data for the complete Evaluation as Scholarly Activity Checklist by AEA Standards.

Program evaluation of in-patient treatment units for adults with acquired brain injury and challenging behavior

ABSTRACT Rationale As acquired brain injury rates continue to increase, the ongoing need for efficient and effective treatment within neurobehavioral rehabilitation settings is clear. Some evidence suggests certain treatment components may be very important to incorporate into service delivery models (e.g., multidisciplinary). However, program evaluation literature and the uptake of complementary intervention strategies, like applied behavior analysis (ABA), in existing neurobehavioral settings remains largely unexplored. Primary Objective The purpose of this project was to: (1) develop and implement a simple, systematic program evaluation informed by best-practices (i.e., research) to assess service delivery models of several neurobehavioral rehabilitation settings, and (2) survey the current use of ABA by participating neurobehavioral agencies. Methodology The program evaluation tool was applied to the charts of randomly selected past and current clients (referred to as participants). A secondary research assistant independently reviewed 29% of the charts to conduct interobserver agreement, which s. was 80% (range, 53%-100%) Results Average program evaluation total percentage score was 33% (range, 4% – 63%), and program evaluation items describing ABA-uptake suggested the incorporation of ABA was low. Discussion We discuss service model areas of strengths and areas for improvement as specified by tool outcomes, as well as in relation to quality improvement implications.

Measuring sexual violence stigma in humanitarian contexts: assessment of scale psychometric properties and validity with female sexual violence survivors from Somalia and Syria

BackgroundValid measures of sexual violence stigma that can be readily incorporated into program monitoring and evaluation systems are needed to strengthen gender-based violence (GBV) services in humanitarian emergencies. This study sought to assess the psychometric properties, construct validity, and measurement invariance of sexual violence stigma scales among female Somali GBV survivors in Kenya and Syrian GBV survivors in Jordan to identify an abbreviated scale that could be used across humanitarian contexts.MethodsWe administered measures of sexual violence stigma to 209 female survivors of sexual violence aged 15 and older in Kenya and Jordan. Exploratory factor analysis was used to assess the underlying latent structure, and Item Response Theory was used to estimate item difficulty and discrimination parameters to guide efforts to shorten the scales. Differential item functioning (DIF) by site was assessed using Multiple Indicators, Multiple Causes models. Construct validity of the sexual violence stigma scales was assessed by estimating correlations with functional impairment, depression, and disability.ResultsThe sexual violence stigma measure exhibited distinct factor structures among Somali and Syrian GBV survivors. Among Somali survivors, a two-factor model with separate felt (10 items) and enacted (4 items) stigma constructs was identified, with scales for both domains exhibiting good internal consistency (Cronbach’s alpha = 0.93 and 0.88, respectively). In Jordan, a single factor solution was uncovered for a 15-item stigma scale with good internal consistency (alpha = 0.86). The shortened core sexual stigma scale consisting of the 4 items that did not exhibit DIF had a Cronbach’s alpha of 0.82 in Kenya and 0.81 in Jordan. The felt stigma scale in Kenya, the full stigma scale in Jordan, and abbreviated core stigma scales in both countries were meaningfully correlated with depression, while correlations with functional impairment were weaker and inconsistent across scales.ConclusionsAn abbreviated core set of invariant perceived and internalized sexual violence stigma items demonstrated evidence of construct validity in two diverse settings. The ability of this measure to be efficiently administered as a part of routine program monitoring and evaluation activities, with the potential addition of items from a measurement bank to improve contextual relevance, can facilitate improvements in the delivery and quality of gender-based violence programs in humanitarian emergencies.

Evaluation of a person‐centered electronic health record care plan for hospitalized patients with dementia

AbstractBackgroundHealth care for people with dementia (PWD) has been characterized as inconsistent in quality and fraught with potentially preventable encounters to hospitals (Petty et al., 2019; Phelan et al., 2012). Person‐centered care (PCC) is the preferred model of dementia care because it improves the quality of care and quality of life, yet acute hospital settings struggle to provide care that values personhood (Clissett et al., 2013).ObjectiveThis DNP scholarly project evaluated the effectiveness of the recently expanded, EHR‐embedded, Patient‐specific Dementia Information (PSDI) care plan and aimed to understand its interaction on nurse perception of dementia care, nurse knowledge of PSDI, and patient‐level outcomes guided by the Donabedian Model to measure its impact on enhancing dementia care delivery at a quaternary medical center.MethodNonexperimental, systematic program evaluation design. Pre‐ and post‐implementation survey measured the general attitudes of acute hospital nurses in providing care to persons with dementia, the extent to which they perceived their clinical practice as being person‐centered, and their general knowledge of the EHR tool. Weekly rounding assessed facilitators and barriers to individualizing patient care based on information extracted from the dementia care plan tool. EHR chart examined primary patient‐level outcome of dementia care quality in the acute care setting (length of stay, restraint use, constant observation, fall occurrence, and superimposed delirium). ResultsThe findings illustrated reduction in fall occurrence, restraint use, and an increase in positive Confusion Assessment Method delirium screening. Though results from nurse survey demonstrated high level of perceived confidence in dementia care, thematic analysis revealed poor confidence in evaluating psychosocial needs. Thematic analysis and survey responses elucidated bedside coaching as in effective strategy to promote staff awareness and learning of the PSDI and PCC practices.ImplicationsIncorporation of PCC by hospital nurses through an EHR‐based care plan improves outcomes for hospitalized adults with dementia and reduce clinical complications. Further advancement can be made by utilizing the PSDI tool during care transition, offering robust education on dementia care, tailored bedside coaching techniques to teach concrete PCC intervention, promoting interdisciplinary involvement in PSDI workflow and future QI initiatives to improve care quality for PWD.

Using standardized exams for formative program evaluation

Systematic program evaluation is important for assessing nursing program quality and achievement of outcomes. Both summative and formative program evaluation have unique value and purpose, but most program evaluation literature to date has focused on summative methods. The purpose of this article is to describe a formative program evaluation method using standardized exam data that was established by an assessment and evaluation committee in a pre-licensure baccalaureate nursing program during the COVID-19 pandemic. Suggestions for how to use the results to support student achievement of program learning outcomes and program-level attainment of benchmarks are discussed.