Program Evaluations Research Articles

Beyond conventional teaching towards networked learning

With Generative Artificial Intelligence (GenAI) adoption growing, education has seen the emergence of innovative technologies like chatbots. However, little research has examined the impacts of GenAI integration in specialized higher education contexts. This study explored graduate students’ experiences using a GenAI chatbot, PEARL, within a graduate-level teacher education course focused on teaching students how to collaboratively conduct program evaluations using practice cases. Four students participated in our study and shared perceptions of interviewing personas with PEARL when evaluating the practice cases. Thematic analysis identified advantages like enhanced efficiency and accessibility, plus limitations regarding authenticity of artificial interactions. Findings emphasized the continued importance of human guidance and peer learning to enrich GenAI-enabled education aligning with principles of networked learning. Students highlighted the need for ethical considerations despite interacting with artificial entities, underscoring nuanced understanding. The significance of collaborative analysis and ongoing iterative improvements also emerged as themes integral to meaningful learning. Although GenAI presents transformational potential in instructional designs, findings support the use of blended approaches that strategically integrate its advantages with human activity and collaborative inquiry. The study makes contributions by elucidating domain-specific nuances of integrating GenAI into teaching in higher education. Practical implications encourage scaffolding GenAI curricula to promote authenticity and collaborative knowledge construction. Further research could examine variations across disciplines, technologies, and demographics. Overall, as GenAI shapes academia’s evolution, reflective pedagogical examination will be key to evidence-guided integration. This exploratory study presents a preliminary yet important step, unveiling opportunities for networked learning and complexities of GenAI adoption in teaching program evaluation skills in education contexts.

Networked Educational Design for Novice Principals’ Professional Development

This paper focuses on professional development for school principals, drawing on literature on the education of leaders and school leaders' development. The Swedish National Principal Training Programme (NPTP) is a state-regulated executive education for professional principals in preschools, schools, and school-age Educare centres. It has been mandatory for all newly appointed principals in Sweden since 2010. The NPTP is a mandatory in-service programme and consists of three courses: school law and governance, governing with objectives and results, and school leadership. The programme runs over three years, parallel to working full-time as a principal. The participants are expected to spend 20% of their working time on their studies. The NPTP uses a networked educational design, which is a valuable research topic from a networked professional learning perspective. This pilot study aims to explore the role of networked educational design for novice principals' professional development and strengthened professional confidence and identity. The research question that frames the study is: In what ways do novice principals' experiences of professional development contribute to the development of their professional confidence and identity with a networked educational design? The study employed a qualitative case-study design with a purposive sampling method to include novice principals undertaking Sweden's national school training programme. Data were collected during 2018-2022 from post-programme evaluations, including individual digital questionnaires with close-ended and open-ended responses and small group oral evaluations documented in a log book. The participants were novice principals studying in the NPTP at one university in Sweden. In all, 321 school leaders from 10 cohort groups in the programme consisting of principals and deputy principals from preschool, primary, secondary, and adult education participated voluntarily. The programme evaluations were carried out during the participants' last residential stay in the programme. Results indicate the importance of residential stays and networking opportunities in the educational journey of principals. These opportunities help connect with peers, establish contacts, and acquire valuable strategies. Completing the training increased principals' professional confidence, particularly in their roles as school leaders. The study shows that the networked educational design supports the development of a professional network among principals, which enhances their professional identity and confidence. The COVID-19 pandemic has highlighted the importance of this networked educational design, which helps shape principals' professional identity and confidence. The findings emphasise the value of networked professional learning.

디지털 리터러시 성인학습에서 교육 형식성이 디지털 역량개발에 미치는 영향

This study investigates the impact of educational formality on developing dig ital competency, a key outcome of digital literacy learning for adults, which has become increasingly popular. We surveyed 151 adult learners with experience in digital literacy learning to assess their pursuit of formal learning structures and measured their resulting digital capabilities. Multiple regression analysis performed using SPSS 27.0 software revealed a positive impact of educational formality on digital competency development. However, qualitative aspects of formality, such as step-by-step continuous learn ing and the exploration of related content, were shown to exhibit greater influence than quantitative aspects such as program structure, professional instructors, and evaluations. These findings suggest that adult digital literacy programs operated by educational institutions should stress qualitative formality to foster adult learners‘ continuous improvement and knowledge growth.

Improving Indigenous health equity within the emergency department: a global review of interventions

IntroductionIndigenous health equity interventions situated within emergency care settings remain underexplored, despite their potential to influence patient care satisfaction and empowerment. This study aimed to systematically review and identify Indigenous equity interventions and their outcomes within acute care settings, which can potentially be utilized to improve equity within Canadian healthcare for Indigenous patients.MethodsA database search was completed of Medline, PubMed, Embase, Google Scholar, Scopus and CINAHL from inception to April 2023. For inclusion in the review, articles were interventional and encompassed program descriptions, evaluations, or theoretical frameworks within acute care settings for Indigenous patients. We evaluated the methodological quality using both the Joanna Briggs Institute checklist and the Ways Tried and True framework.ResultsOur literature search generated 122 publications. 11 articles were selected for full-text review, with five included in the final analysis. Two focusing on Canadian First Nations populations and three on Aboriginal Australians. The main intervention strategies included cultural safety training, integration of Indigenous knowledge into care models, optimizing waiting-room environments, and emphasizing sustainable evaluation methodologies. The quality of the interventions was varied, with the most promising studies including Indigenous perspectives and partnerships with local Indigenous organizations.ConclusionsAcute care settings, serving as the primary point of access to health care for many Indigenous populations, are well-positioned to implement health equity interventions such as cultural safety training, Indigenous knowledge integration, and optimization of waiting room environments, combined with sustainable evaluation methods. Participatory discussions with Indigenous communities are needed to advance this area of research and determine which interventions are relevant and appropriate for their local context.

Perioada nord-americană

The purpose of this article is to bring to the forefront the theatrical activity carried out for eight years by Theodor-Cristian Popescu in the North American cultural landscape. Following the path of the Romanian theatre director, the article presents the performances he directed during his master’s degree at the University of Montana and also the theatrical events he staged in Montréal and Toronto, once he moved to Canada. Drawing on a variety of sources (press articles – reviews, interviews, press releases –, as well as program booklets, grant applications or evaluations of his teaching activity), the article aims to provide the Romanian public with a detailed presentation of Theodor-Cristian Popescu’s theatrical activity abroad (theatrical animator, director, teacher).

Process Evaluations of Diabetes Self-Management Programs: A Systematic Review of the Literature.

To conduct a systematic review of process evaluations (PEs) of diabetes self-management programs (DSMPs). An electronic search using Medline (Ovid), Embase (Ovid), CINAHL (Ensco), Academic Search (Ebsco), and APA PsycInfo (Ebsco). Peer-reviewed, empirical quantitative, qualitative, or mixed-method studies were included if they (1) were a traditional, group-based DSMP, (2) involved adults at least 18 years with T1DM or T2DM, (3) were a stand-alone or embedded PE, and (4) published in English. The following process evaluation outcomes were extracted: fidelity, dose delivered, dose received, reach, recruitment, retention, and context. Additional items were extracted, (eg, process evaluation type, data collection methods; theories; frameworks or conceptual models used to guide the process evaluation, and etc). Due to heterogeneity across studies, studies were synthesized qualitatively (narratively). Sixty-eight studies (k) in 78 articles (n) (k = 68; n = 78) were included. Most were mixed methods of low quality. Studies were typically integrated into outcome evaluations vs being stand-alone, lacked theoretical approaches to guide them, and incorporated limited outcomes such as dose received, reach, and retention. Future research should 1) implement stand-alone theoretically grounded PE studies and 2) provide a shared understanding of standardized guidelines to conduct PEs. This will allow public health practitioners and researchers to assess and compare the quality of different programs to be implemented.

Evaluating the Delivery and Oversight of State-Funded Local Bridge Projects

State-funded local bridge projects face several challenges including but not limited to constrained budgets, increasing burden from state departments of transportation (DOT) staff, and the lack of expertise of some local personnel. By streamlining the delivery and oversight of state-funded projects through delegation, a collaborative and proactive approach can be achieved, ensuring the successful maintenance and development of local low-risk bridges, which are not on the National Highway System and have minimal to no environmental, right of way, utility, or railroad impacts. This paper offers a practical approach to evaluating programs focused on progressing project delivery by delegating specific tasks and responsibilities to local governments. Taken as a study case, Wisconsin Department of Transportation’s low-risk bridge pilot program was evaluated using both a quantitative methodology: data analysis using a data-driven statistical approach involving analysis of pilot and control group data, and an evaluation of the performance metrics developed by the state DOT; and a qualitative approach: a survey of local sponsors and stakeholders about the pilot project. The survey corroborated the findings derived from the data analysis: improved performance in project cost and schedule for pilot projects, highlighting consistency and reliability in the evaluation process. The integration of both methodologies resulted in valuable insights and experiences, providing valuable guidance to other states conducting similar program evaluations.

Cost-effectiveness of Neonatal Hearing Screening Programs: Systematic Review.

Introduction Universal newborn hearing screening (UNHS) has been widely and strongly advocated as an early detection strategy for hearing loss in children. This intervention aims to prevent delays in speech and language development, which, in turn, has long-term beneficial effects on the social and emotional development and quality of life of individuals. However, the implementation of UNHS programs is circumstantial in different settings, for different reasons. Objectives The present systematic review aimed to identify whether the implementation of UNHS programs are cost-effective, as well as their variations by localities. Data Synthesis A search was conducted in seven databases: PubMed (Medline), Scopus, Web of Science, Embase, CINAHL, Lilacs, and Cochrane Library. Studies that included a cost analysis of UNHS programs were eligible for inclusion. Studies on evaluations of preschool or school-based programs only were excluded, among others. A total of 1,291 records were found. Of these, 23 articles were analyzed in full. All articles identified the cost-effectiveness of the UNHS programs implemented. Regarding the UNHS protocols, a wide variation was observed in all aspects: tests used, period established between tests and retests, professionals responsible for screening, environment, and criteria for defining hearing loss, limiting the generalization of this information. All studies presented values related to the expenses with the program, but none of them showed statistical elements for the described analyzes or any theoretical basis for such. Conclusion It is necessary to estimate local specific issues, as well as the accuracy of the chosen tests and the NHS protocols used, so that more accurate analyzes on cost-effectiveness are possible.

Culinary Medicine Experiences for Medical Students and Residents in the U.S. and Canada: A Scoping Review

Phenomenon Despite the importance of diet in the prevention and management of many common chronic diseases, nutrition training in medicine is largely inadequate in medical school and residency. The emerging field of culinary medicine offers an experiential nutrition learning approach with the potential to address the need for improved nutrition training of physicians. Exploring this innovative nutrition training strategy, this scoping review describes the nature of culinary medicine experiences for medical students and resident physicians, their impact on the medical trainees, and barriers and facilitators to their implementation. Approach This scoping review used the Joanna Briggs Institute methodology for scoping reviews and the Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for Scoping Reviews (PRISMA-ScR) checklist as guides. Eligible publications described the nature, impact, facilitators, and/or barriers of nutrition and food preparation learning experiences for medical students and/or residents. Additional inclusion criteria were location (U.S. or Canada), allopathic or osteopathic, English, human subjects, and publication year (2002 or later). The search strategy included 4 electronic databases. Two reviewers independently screened titles/abstracts and a third reviewer resolved discrepancies. The full-text review consisted of 2 independent reviews with discrepancies resolved by a third reviewer or by consensus if needed, and the research team extracted data from the included articles based on the nature, impact, barriers, and facilitators of culinary medicine experiences for medical trainees. Findings The publication search resulted in 100 publications describing 116 experiences from 70 institutions. Thirty-seven publications described pilot experiences. Elective/extracurricular and medical student experiences were more common than required and resident experiences, respectively. Experiences varied in logistics, instruction, and curricula. Common themes of tailored culinary medicine experiences included community engagement/service-based learning, interprofessional education, attention to social determinants of health, trainee well-being, and cultural considerations. Program evaluations commonly reported the outcome of experiences on participant attitudes, knowledge, skills, confidence, and behaviors. Frequent barriers to implementation included time, faculty, cost/funding, kitchen space, and institutional support while common facilitators of experiences included funding/donations, collaboratives and partnerships, teaching kitchen access, faculty and institutional support, and trainee advocacy. Insights Culinary medicine is an innovative approach to address the need and increased demand for improved nutrition training in medicine. The findings from this review can guide medical education stakeholders interested in developing or modifying culinary medicine experiences. Despite barriers to implementation, culinary medicine experiences can be offered in a variety of ways during undergraduate and graduate medical education and can be creatively designed to fulfill some accreditation standards.

A Global Perspective on Oral Vaccination of Wildlife against Rabies.

The long-term mitigation of human-domestic animal-wildlife conflicts is complex and difficult. Over the last 50 yr, the primary biomedical concepts and actualized collaborative global field applications of oral rabies vaccination to wildlife serve as one dramatic example that revolutionized the field of infectious disease management of free-ranging animals. Oral vaccination of wildlife occurred in diverse locales within Africa, Eurasia, the Middle East, and North America. Although rabies is not a candidate for eradication, over a billion doses of vaccine-laden baits distributed strategically by hand, at baiting stations, or via aircraft, resulted in widespread disease prevention, control, or local disease elimination among mesocarnivores. Pure, potent, safe, and efficacious vaccines consisted of either modified-live, highly attenuated, or recombinant viruses contained within attractive, edible baits. Since the late 1970s, major free-ranging target species have included coyotes (Canis latrans), foxes (Urocyon cinereoargenteus; Vulpes vulpes), jackals (Canis aureus; Lupulella mesomelas), raccoons (Procyon lotor), raccoon dogs (Nyctereutes procyonoides), and skunks (Mephitis mephitis). Operational progress has occurred in all but the latter species. Programmatic evaluations of oral rabies vaccination success have included: demonstration of biomarkers incorporated within vaccine-laden baits in target species as representative of bait contact; serological measurement of the induction of specific rabies virus neutralizing antibodies, indicative of an immune response to vaccine; and most importantly, the decreasing detection of rabies virus antigens in the brains of collected animals via enhanced laboratory-based surveillance, as evidence of management impact. Although often conceived mistakenly as a panacea, such cost-effective technology applied to free-ranging wildlife represents a real-world, One Health application benefiting agriculture, conservation biology, and public health. Based upon lessons learned with oral rabies vaccination of mesocarnivores, opportunities for future extension to other taxa and additional diseases will have far-reaching, transdisciplinary benefits.

Food Systems Interventions for Nutrition: Lessons from 6 Program Evaluations in Africa and South Asia

Although there is growing global momentum behind food systems strategies to improve planetary and human health—including nutrition—there is limited evidence of what types of food systems interventions work. Evaluating these types of interventions is challenging due to their complex and dynamic nature and lack of fit with standard evaluation methods. In this article, we draw on a portfolio of 6 evaluations of food systems interventions in Africa and South Asia that were intended to improve nutrition. We identify key methodological challenges and formulate recommendations to improve the quality of such studies. We highlight 5 challenges: a lack of evidence base to justify the intervention, the dynamic and multifaceted nature of the interventions, addressing attribution, collecting or accessing accurate and timely data, and defining and measuring appropriate outcomes. In addition to more specific guidance, we identify 6 cross-cutting recommendations, including a need to use multiple and diverse methods and flexible designs. We also note that these evaluation challenges present opportunities to develop new methods and highlight several specific needs in this space.

Differential Intervention Outcomes Among Fathers Who Commit Domestic and Family Violence: The Influence of Parental Relationship Status

Abstract Background In Western contexts, the behavior of domestically violent men is largely addressed through legal sanctions and standardised men’s behavior change programs. When domestically violent fathers remain together or in contact with their families, safety is an ongoing concern. However, program evaluations have failed to provide clear evidence for their effectiveness in addressing family violence in relation to parenting status. Part of the problem in developing efficacious programs has been the challenge of targeting them to typologies or socio-cultural needs of participants. Purpose The Caring Dads (CD) program supports father-change through leveraging men’s motivation to be positive parents, promoting child-centred fathering and cessation of family abuse. The aim of this study was to investigate whether participating fathers improved their co-parenting interactions, family functioning and wellbeing, and considered father characteristics and relationship statuses that coincided with positive outcomes. Methods This mixed method Australian study investigating a CD pilot incorporated psychological measures and semi-structured interviews at pre-, post- and 10 months post-intervention. Participants were fathers who attended CD in South-East Queensland, 2017–2019, and their co-parents. Results All fathers experienced improvements in parenting and family functioning by post-intervention, with greatest improvements for partnered fathers’ families. Partnered fathers also experienced improved parental alliance, unlike most separated fathers, although results must be viewed with caution due to small sample size and attrition across the study duration. Conclusion This study provides early support for improvements in CD participants’ fathering, parental alliance, and family functioning, most substantially for partnered fathers, though findings should be further investigated with larger cohorts.

Using theory knitting to conceptualize social phenomena in the design and evaluation of dance programs for people living with Parkinson’s disease

Dance programs for people living with Parkinson’s disease (PwPD) offer participants an opportunity to exercise, engage in artistic self-expression, and form new relationships. While it is understood that the social dimension of dance programs for PwPD contributes to dancer satisfaction and program sustainability, the social mechanisms instrumental to program success are under-examined. Engaging with theory from wider disciplines, or “theory knitting” can help program designers and evaluators examine the mechanisms and contextual factors that make classes socially meaningful with greater detail and specificity. This study identified and examined three theoretical frameworks that program planners and evaluators could use to conceptualize social engagement in dance for PwPD contexts and inform practice. Each theory was assessed for fit using the T-CaST theory comparison and selection tool developed by Birken et al. (2018). As an example, we used anthropologist Victor Turner’s (1970; 1977) theory of liminality and communitas to identify five key areas for fostering a sense of social connection in dance for PwPD contexts: (1) selecting a meaningful dance space (2) creating a joyous atmosphere (3) marking entrance into the liminal time and space with rituals (4) embodying liminality and anti-structure and (5) inverting power relations and embracing fluid roles.

Analyzing Student Success Outcome Variables in Higher Education Utilizing the Chi-Square Test of Independence

For the past two decades student success measures such as student persistence, retention, and graduation rates have been a point of emphasis in higher education. These measures are often directly related to funding formulas for state public colleges and universities. Therefore, analyses of these data have become more critical to evaluating student success initiatives for faculty and administration at many institutions. However, while these data are often widely available there is very little higher education research on how they should be analyzed to assess student success initiatives, program evaluations, or teaching effectiveness at the institutional level.As student success outcome variables are categorical in nature, linear analyses of these data may prove rather difficult as a dependent variable without a significant amount of transformation. Therefore, the purpose of this article is to provide practitioners with a simple, yet powerful option for analyzing student success outcome variables utilizing the Chi-square test of independence. A case study approach was taken to illustrate how Chi-square can be used to specifically analyze the association between an experiential learning high impact practice and graduation rates among undergraduate students. This case was based on a results and interpretation perspective, rather than step-by-step instruction on how to perform the analysis itself.

Effectiveness and Feasibility of virtual international exchange program for occupational therapy students to develop the international perspective and professional skills -Mix-method study

Background: Virtual international exchange programs have been promoted; however, the effectiveness of the programs has not yet been examined well in the field of occupational therapy (OT) field. This study aimed to examine the effectiveness and feasibility of a virtual international exchange program for OT students.Methods: Mix-method was applied in this study.Two universities from Japan and Indonesia created an exchange program to cultivate international perspectives and facilitate gaining OT professional knowledge and skills, as well as learning basic scientific study skills. Questionnaires on international minded and program evaluations, which were analyzed by descriptive analysis, were given to the participants before and after the program. Focus group interviews, which were analyzed with reflective thematic analysis, were conducted after the program.Results: Ten students joined the program, six students answered the questionnaires, and eight students participated in the focus group interviews. Students were satisfied with the program overall, even though there were no significant changes in the total scores of the international minded between before and after the program. In the focus group interviews, the participants mentioned their motivations for participating in the program, program evaluation, international minded, and self-growing awareness. The participants expressed that they gained OT professional knowledge and academic skills from the program.Conclusion: As a feasible virtual program, well-organized programs considering the students’ academic learning skills and interests could provide opportunities for individual development, acquiring professional knowledge, and incite self-growing. In addition, the result might show the effectiveness of virtual programs to cultivate international perspective and professional knowledge.

IMPLEMENTASI PROGRAM TAKHOSSUS LPBA DI PONDOK PESANTREN RAUDLATUL MALIKIYAH

The implementation of the Takhossus LPBA program at the Raudlatul Malikiyah Islamic Boarding School involves developing the Arabic language, teaching Arabic grammar rules, and implementing routine language activities. This program develops students' Arabic language skills, such as speaking, debating, hosting and delivering speeches. In addition, this research explains the history and teaching system of the program, and highlights the importance of nahwu and sharaf in learning Arabic to understand the structure and use of the language. The research methodology used is work analysis using primary and secondary data sources. The LPBA program in Islamic boarding schools is implemented through BANOM dirofiyah activities, which include routine Arabic language rules and sanctions for violators. This research aims to explore the implementation of the LPBA program in Islamic boarding schools using qualitative research methods. LPBA as a foreign language development institution encourages its members to speak Arabic every day to improve their language skills. Apart from that, Islamic boarding schools also implement the Takhassus LPBA program to deepen their knowledge of the Arabic language. Although the program faces challenges such as lack of student motivation resources, limited number of teachers, limited expertise, limited textbooks, teaching materials and multimedia resources, strategic steps have been taken to overcome these problems. Program evaluations are carried out periodically to ensure the effectiveness of activities. Keywords: Takhossus; Arabic; Banom.

Evaluation of undergraduate academic programs through data envelopment analysis and time-to-degree estimates at Spanish public universities

This article proposes nonparametric and parametric methods for conducting meaningful evaluations of academic programs. In particular, it proposes the use of data envelopment analysis (DEA) along with regression analysis for the evaluation of Spanish public universities' undergraduate curricula. Unlike the radial DEA models, which assume an equiproportional expansion of all outputs to achieve efficiency, the current study also relies on the use of non-radial DEA models for assessing the efficiency of Spanish public universities. A non-radial DEA measure (the so-called Russell measure) allows for unlike proportional augmentations in each positive output. In this study, non-radial DEA measures allow us to identify different levels of inefficiency for each output considered (social sciences and non-social sciences degrees). Specifically, the results show that the inefficiency in the production of scientific and technical degrees is greater than that in the production of humanities and social sciences degrees, although there are differences among institutions. Bachelor's degree production time is also estimated, and a duration analysis explains the time to degree. A worrying result is the "excessive" time it takes for a university to produce four-year undergraduate degree programs. The mean graduation time is 5.7 years. The Cox proportional hazards regression shows that a shorter time to graduation is associated with higher teaching quality as well as a higher faculty-to-student ratio. Parametric survival analysis using a lognormal distribution with gamma frailty also verifies these latter results.

Implementation of a Blended-Learning Perioperative Nursing Education Program in Canada.

Governmental COVID-19 mandates in Ontario, Canada, resulted in a backlog of perioperative procedures. Organization leaders were required to expand services after the pandemic; however, the ongoing nursing shortage and college-based structure of perioperative education programs complicated their response. In 2021, we developed an in-house perioperative education program using a blended-learning theory comprising online modules and videos, skills laboratory sessions, and clinical placement experiences. Nurses were required to apply for the program and remain employed at the facility for two years. Program evaluations showed that the novice nurses felt confident when beginning clinical experiences and preceptors believed the nurses were prepared for practice. Sixteen of 19 participants successfully completed the program, which helped resolve the staffing shortage. Novice nurses may benefit from a shadowing experience before applying for this type of program. Leaders in nonperioperative specialties should consider an in-house education program to help meet staffing needs in their areas.

Adopting the RE-AIM analytic framework for rural program evaluation: experiences from the Advance Care Planning via Group Visits (ACP-GV) national evaluation.

To support rigorous evaluation across a national portfolio of grants, the United States Department of Veterans Affairs (VA) Office of Rural Health (ORH) adopted an analytic framework to guide their grantees' evaluation of initiatives that reach rural veterans and to standardize the reporting of outcomes and impacts. Advance Care Planning via Group Visits (ACP-GV), one of ORH's Enterprise-Wide Initiatives, also followed the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. ACP-GV is a national patient-centered intervention delivered in a large, veterans integrated healthcare system. This manuscript describes how RE-AIM was used to evaluate this national program and lessons learned from ORH's annual reporting feedback to ACP-GV on their use of the framework to describe evaluation impacts. We used patient, provider, and site-level administrative health care data from the VA Corporate Data Warehouse and national program management databases for federal fiscal years (FY) spanning October 1, 2018-September 30, 2023. Measures included cumulative and past FY metrics developed to assess program impacts. RE-AIM constructs included the following cumulative and annual program evaluation results. ACP-GV reached 54,167 unique veterans, including 19,032 unique rural veterans between FY 2018 to FY 2023. During FY 2023, implementation adherence to the ACP-GV model was noted in 91.7% of program completers, with 55% of these completers reporting a knowledge increase and 14% reporting a substantial knowledge increase (effectiveness). As of FY 2023, 66 ACP-GV sites were active, and 1,556 VA staff were trained in the intervention (adoption). Of the 66 active sites in FY 2023, 27 were sites previously funded by ORH and continued to offer ACP-GV after the conclusion of three years of seed funding (maintenance). Lessons learned developing RE-AIM metrics collaboratively with program developers, implementers, and evaluators allowed for a balance of clinical and scientific input in decision-making, while the ORH annual reporting feedback provided specificity and emphasis for including both cumulative, annual, and rural specific metrics. ACP-GV's use of RE-AIM metrics is a key step towards improving rural veteran health outcomes and describing real world program impacts.

Strategies to develop an LGBTQIA+-inclusive adolescent sexual health program evaluation.

Adolescent sexual health interventions are increasingly incorporating content that is inclusive of LGBTQIA+ youth (lesbian, gay, bisexual, transgender, queer/questioning, intersex, asexual, and other marginalized sexualities and genders). Evaluations of such programs must also be inclusive to enhance the validity of evaluation results and avoid further marginalization. We present strategies for increasing LGBTQIA+-inclusivity based on our evaluation of SafeSpace, a sexual health curriculum. To design an LGBTQIA+-inclusive program evaluation, we leveraged LGBTQIA+ research staff's insights, pursued a parental consent waiver, developed an inclusive recruitment plan, and crafted demographic and sexual behavior survey measures with input from youth and equity experts. We conducted a pilot study with 42 youth ages 14-17 to assess the feasibility and efficacy of our strategies. We obtained a parental consent waiver and recruited a majority LGBTQIA+ pilot study sample (62%). Using themes from cognitive interviews with youth and experts regarding inclusive framing and use of plain language, we refined demographic measures and expanded sexual behavior measures. Findings suggest that the strategies used to enhance LGBTQIA+-inclusivity in our evaluation of SafeSpace were effective in respectfully and more accurately capturing a fuller range of experiences and identities of LGBTQIA+ and cis-straight youth. The strategies and survey measures developed for this study can be applied to increase LGBTQIA+-inclusivity in other adolescent sexual health program evaluations.