Results Of Program Evaluation Research Articles

A New Comprehensive Methodology for Evaluation of Course Outcomes and Programme Outcomes

Abstract : Course Outcomes (CO) and Programme Outcomes (PO) are the backbone of Outcome Based Education (OBE) and their evaluation using a definite established methodology is necessary to track the progress and effective implementation of OBE. The CO, Course and PO attainment levels determine the extent to which the skill/abilities as envisaged vide POs are imbibed by the students. The CO, Course and PO attainment also provide an insight for corrective actions. A new method for determination of CO, Course and PO attainment level is presented in this paper. The method is comprehensive and unique as it takes into consideration the varying weightage of each course component such as Continuous Evaluation (CE), Semester End Examination (SEE), and Laboratory and Practical Work (LPW). This not only ensures that all assessment components are covered but it also has a provision to take into account the effect of multiplemapping. This allows a teacher the flexibility to frame questions which can map with multiple COs. The proposed method gives individual as well as class attainment levels of the Course and the COs, thus allowing a more focused root cause analysis and corrective actions. The method also allows for calculation of the PO attainment level based on the CO and Course attainment level and the Program Articulation Matrix. The criteria for deciding the actions based on the PO and CO assessment levels are also presented. Keywords: Accreditation; Course outcome; Direct attainment; Outcome based education; Program outcome

Evaluation of Bidikmisi scholarship program at state Islamic universities in Indonesia

This study raised the theme of evaluating the Bidikmisi scholarship program at State Islamic Universities in Indonesia with the CIPP Model approach. The Bidikmisi scholarship program is the government's flagship program for outstanding students who are less able to meet the costs of higher education. This research is evaluation research focusing on four areas: context, input, process, and product evaluation. The research subjects were one managing director, three universities within the Ministry of Religion of the Republic of Indonesia, and 15 student informants who received the Bidikmisi scholarship. The results of the research on the evaluation of the Bidikmisi scholarship program show that (1) the context of the program is carried out in accordance with the technical instructions issued by the directorate of Islamic higher education related to the evaluation of the Bidikmisi scholarship program. (2) Evaluation of inputs is carried out by meeting the criteria considering starting from setting evaluation goals to the results of program evaluation, and program targets are high school graduates who have academic and non-academic achievements. (3) In the current process, the Bidikmisi scholarship program has been running for ten years with satisfactory results, and the selection is made openly and transparently. (4) The product of this program evaluation is policy improvement and service improvement to prospective scholarship recipients. The program manager must maintain its performance by improving program quality, achievement process, and control. Programs should focus on increasing resources by strengthening program recipients' academic skills and knowledge.

간호학 학사학위 프로그램 학습성과 평가체계 개발

Purpose : This study was conducted to develop an program outcome evaluation system for the Utilizing Information and Communications Technology (ICT) and the Latest Health-Care Technology, required for nursing education accreditation. With respect to the accreditation criterion on program outcomes, it is necessary to understand the relationship between nursing education objectives, curriculum, and assessment. Methods : A methodological research design was adopted. To ensure the theoretical validity of the program outcome evaluation system, it was verified against the learning objectives of nursing education programs and a domestic literature review in Korea. The program outcome evaluation system for the Utilizing Information and Communications Technology (ICT) and the Latest Health-Care Technology Program outcome evaluation system, based on the Kim & Park’s developmental model (2008), was then developed. Results : The program outcome evaluation system was established, including implementation level, educational curriculum, performance level, evaluation method, rubrics, and closing-the-loop processes. Conclusion : The finalized program outcome evaluation system can be actively used in nursing education and contribute to enhancing the competency of nursing students and graduates in utilizing ICT and health-care technology.

민간부문 긴급복지지원사업 사례연구

Emergency assistance program (EAP) is not only operated separately from public welfare system for the low-income vulnerable, but also distinguished from disaster management support program, which is designed to respond to natural disasters such as hurricane, tornado, and COVID-19, etc. EAP is usually targeting to support all vulnerable populations at crisis, who are on the borderline of blind spots in public welfare system. In addition to public sector, private sector has also carried out EAPs. However, little research has been conducted in this field. The purpose of this study is to analyze the outcome of A Foundation’s ‘SOS Emergency Welfare Support Program’, and deduce explanatory implications. Thus, this study is a kind of program outcome evaluation research. The research method consists of content analysis and related literature review. As data sources, this study used two types of reports: the ‘On-site Verification Report’ and ‘Support-Result Report’ for the program. After analyzing the contents of those reports, the cases were quantitatively converted for analysis. A total of 657 cases were analyzed for three years, from 2018 to May 2021. One of findings is that economic and housing problems were the main reasons for crisis. Many cases had complex problems such as mental health issues and domestic violence. As for performance indicators, problem-centered target efficiency, client-centered target efficiency, contribution to problem-solving, speed of the process, and positive side effects were measured. They present the program has positive outcomes. Based on the results, the future direction of EAP has been discussed even in relation to the public sector.

Impactability Modeling for Reducing Medicare Accountable Care Organization Payments and Hospital Events in High-Need High-Cost Patients: Longitudinal Cohort Study.

BackgroundImpactability modeling promises to help solve the nationwide crisis in caring for high-need high-cost patients by matching specific case management programs with patients using a “benefit” or “impactability” score, but there are limitations in tailoring each model to a specific program and population.ObjectiveWe evaluated the impact on Medicare accountable care organization savings from developing a benefit score for patients enrolled in a historic case management program, prospectively implementing the score, and evaluating the results in a new case management program.MethodsWe conducted a longitudinal cohort study of 76,140 patients in a Medicare accountable care organization with multiple before-and-after measures of the outcome, using linked electronic health records and Medicare claims data from 2012 to 2019. There were 489 patients in the historic case management program, with 1550 matched comparison patients, and 830 patients in the new program, with 2368 matched comparison patients. The historic program targeted high-risk patients and assigned a centrally located registered nurse and social worker to each patient. The new program targeted high- and moderate-risk patients and assigned a nurse physically located in a primary care clinic. Our primary outcomes were any unplanned hospital events (admissions, observation stays, and emergency department visits), count of event-days, and Medicare payments.ResultsIn the historic program, as expected, high-benefit patients enrolled in case management had fewer events, fewer event-days, and an average US $1.15 million reduction in Medicare payments per 100 patients over the subsequent year when compared with the findings in matched comparison patients. For the new program, high-benefit high-risk patients enrolled in case management had fewer events, while high-benefit moderate-risk patients enrolled in case management did not differ from matched comparison patients.ConclusionsAlthough there was evidence that a benefit score could be extended to a new case management program for similar (ie, high-risk) patients, there was no evidence that it could be extended to a moderate-risk population. Extending a score to a new program and population should include evaluation of program outcomes within key subgroups. With increased attention on value-based care, policy makers and measure developers should consider ways to incorporate impactability modeling into program design and evaluation.

Evaluation of postgraduate rural medical training programs and implications for rural workforce development: a systematic review.

Providing postgraduate rural training programs has been recognised as central for successful recruitment and retention of medical workforce. While there have been many documented cases of rural training program development, documented program outcome evaluations are few. This review investigated how postgraduate training and educational programs for rural and remote medicine are evaluated worldwide. Through the use of a systematic review, the study explored three questions: 'What are the outcomes of postgraduate rural and remote training programs worldwide?', 'How are the program evaluations conducted?' and 'What evaluation models and approaches are used in evaluating the effectiveness of these training programs?' A qualitative synthesis was undertaken of evaluations of postgraduate rural training programs published in the English language in medical education journals. The study involved pooling quantitative and mixed-methods research data and findings from qualitative studies, which were aggregated, integrated and interpreted. PubMed, PsycINFO, ERIC and Web of Science databases were searched to identify studies that satisfy the search criteria. Of the 1297 articles identified through the database search, 26 studies were included in the analysis. Most of the evidence from the studies consists of descriptive studies with some longitudinal tracking programs and cohort studies. Nine themes were identified: practice location after training completion; training location and decentralised model; educational aspects; incentives, political contexts and regulations; personal, social and cultural issues; professional development; rural orientation and community engagement; support system; and gender and racial issues. Key outcomes were analysed and cross-validated against the 2020 WHO guideline on health workforce development, attraction, recruitment and retention in rural and remote areas. These studies' most frequent evaluation methods were surveys, followed by interviews, questionnaires and secondary data from existing databases. Methodological characteristics, the relationship between rural background and program outcomes, and implications for decentralised training, telehealth and tele-assessment during the COVID-19 pandemic are discussed. Analysis from the key outcomes suggests evaluation as a strategy to uncover outcomes in postgraduate rural and remote training and medical education. Regardless of the similar outcomes, the program evaluations implemented and the robustness of evidence vary across programs and medical schools. The absence of solid evaluation designs and their alignment to the program objectives will lessen the strengths of evidence. Better quality research and evaluation design, objective settings, qualitative inquiry to uncover the contexts, and developing appropriate indicators and benchmarks for monitoring and evaluating strategies must be considered during program development and implementation.

Evaluating Community-Based Physical Fitness Programs: A Decision-Focused Approach

Compelling evidence supports that Community-based Physical Fitness Programs (CPFP) can improve physical fitness and health. However, a central issue in CPFP is the lack of evaluation data from participating communities. The study aimed to evaluate the CPFP of three local communities using Context, Input, Process, Product (CIPP) model. Using a validated tool with a 4.81 mean (SD=0.38) and substantially agreed by three (3) experts (k=0.76), evaluation data were obtained from community members (n=117). This research primarily employed descriptive and correlation analyses to determine the relationships among CIPP elements and demographic profiles of one highly rated CPFP. 117 community members (28% male; 72% female) were evaluated the CPFP of 3 local communities. The context evaluation results showed that only SJC CPFP had a relatively high result with a mean of 4.29 (85.8%). For input evaluation, SJC still had the highest mean of 4.23 (84.6%), as compared to SCM 3.93 (78.6%) and PC 3.67 (73.4%). SJC remained at the top in the process evaluation, with the highest mean of 4.30 (86%). While on product evaluation, all communities had close mean scores of 4.4 (88%) SJC, 3.9 (78%) SCM, and 4.1 (82%) PC. Correlations among data revealed that SJC context and input evaluation results had a significant relationship with sex and age. There was meager participation of males compared to females and even young members compared to older members. While CPFP among communities received good evaluation results, there is still a need to reinforce the program due to lack of pre-implementation assessment and integration of vision-mission, goals, and objectives (context). There should also be a re-structuring of precise action plans (input), a variety of activities (process), and consistent program outcomes evaluation (product). Lastly, the correlated evaluation data suggested the need for a more personalized, gender-sensitive, and age-friendly CPFP. This study provides CPFP administrators with comprehensive data that can assist them in making sound decisions for program improvement.

Manajemen Strategi Pondok Pesantren dalam Meningkatkan Kompetensi Santri

This study aims to understand the strategic management of Al-Ma’soem Students Islamic Boarding School in increasing students’ competence starting from formulation, implementation and evaluation of a qualitative descriptive approach as an alternative method of this interviews, observation, and documentation studies. From the research it can be concluded that the Al-Ma’soem Islamic Boarding School students have used a good management strategy. This can be seen from the proven strategy formulation with the formulation of strategies composed of vision, mission, and environmental analysis which are then applied into a strategy. Second, the implementation of the strategy in the ferm of sereval programs and activities arranged by the curriculum and adapted to the needs of students. Third, Strategy evaluation is carried out with two techniques namely program evaluation techniques and outcome evaluation. The final result of this evaluation aim to be able to know the effectiveness of a program that has been designed and implemented in order to improve the competence of students.

Evaluation of a carbapenem antimicrobial stewardship program and clinical outcomes in a Japanese hospital

IntroductionMicroorganisms can evolve and become resistant to antimicrobials, and this is known as antimicrobial resistance (AMR). Inappropriate use of antibiotics contributes to AMR, and antimicrobial stewardship programs have been developed to mitigate AMR. The Appropriate Use of Carbapenems Program was implemented in March 2019 in a university hospital and its effect was evaluated. MethodsWe conducted a prospective audit and feedback on carbapenems at the time of prescription daily. Additionally, we compared a monthly survey of the total days of therapy (DOTs) per 1000 patient-days for carbapenems, piperacillin/tazobactam, and fluoroquinolones. The susceptibility of Pseudomonas aeruginosa to meropenem, piperacillin/tazobactam, and levofloxacin was tested before (January 2018 to February 2019) and after (March 2019 to December 2020) the intervention. ResultsThe monthly median DOTs of carbapenem usage decreased after the intervention; carbapenem use immediately declined during the intervention period. The monthly median DOTs of piperacillin/tazobactam and fluoroquinolones also decreased and continued to decline significantly after the intervention. Susceptibility of P. aeruginosa to meropenem, piperacillin/tazobactam, and levofloxacin did not change significantly during the study. ConclusionThe implementation of the Appropriate Use of Carbapenems Program was effective in reducing the use of broad-spectrum antibiotics and maintaining the antibiotic susceptibility of P. aeruginosa.

Professionalism in Family Planning Care Workshop.

IntroductionWhen clinicians feel negative emotions toward patients, providinge patient-centered care can be difficult. This can occur in family planning scenarios, such as when a provider is uncomfortable with a patient choosing abortion. The Professionalism in Family Planning Care Workshop (PFPCW), framed around professionalism values, used guided reflection to foster self-awareness and empathy in order to teach future providers to provide patient-centered care.MethodsIn the PFPCW, learners discussed challenging patient interactions and family planning scenarios to develop self-awareness and identify strategies for maintaining therapeutic relationships with patients when they experience negative feelings toward them. We implemented the workshop across the United States and Canada and collected pre- and postsurvey data to evaluate program outcomes at Kirkpatrick evaluation levels of participant reaction and effects on learners’ attitudes.ResultsA total of 403 participants participated in 27 workshops in which pre- and postworkshop surveys (70% and 46% response rates, respectively) were administered. Sixty-five percent of the participants were residents, and 36% had previously participated in a similar workshop. The majority (92%) rated the PFPCWs as worthwhile. Participants valued the discussion and self-reflection components. Afterward, 23% reported that their attitudes toward caring for people with unintended pregnancy changed to feeling more comfortable. Participants said they would employ self-reflection and empathy in future challenging interactions.DiscussionIn this pilot implementation study, our workshop provided learners with strategies for patient-centered care in challenging family planning patient interactions. We are currently modifying the workshop and evaluation program based on feedback.

\u201cWe are not stray leaves blowing about in the wind\u201d: exploring the impact of Family Wellbeing empowerment research, 1998\u20132021

BackgroundAn Aboriginal-developed empowerment and social and emotional wellbeing program, known as Family Wellbeing (FWB), has been found to strengthen the protective factors that help Indigenous Australians to deal with the legacy of colonisation and intergenerational trauma. This article reviews the research that has accompanied the implementation of the program, over a 23 year period. The aim is to assess the long-term impact of FWB research and identify the key enablers of research impact and the limitations of the impact assessment exercise. This will inform more comprehensive monitoring of research impact into the future.MethodsTo assess impact, the study took an implementation science approach, incorporating theory of change and service utilisation frameworks, to create a logic model underpinned by Indigenous research principles. A research impact narrative was developed based on mixed methods analysis of publicly available data on: 1) FWB program participation; 2) research program funding; 3) program outcome evaluation (nine studies); and 4) accounts of research utilisation (seven studies).Results Starting from a need for research on empowerment identified by research users, an investment of $2.3 million in research activities over 23 years produced a range of research outputs that evidenced social and emotional wellbeing benefits arising from participation in the FWB program. Accounts of research utilisation confirmed the role of research outputs in educating participants about the program, and thus, facilitating more demand (and funding acquisition) for FWB. Overall research contributed to 5,405 recorded participants accessing the intervention. The key enablers of research impact were; 1) the research was user- and community-driven; 2) a long-term mutually beneficial partnership between research users and researchers; 3) the creation of a body of knowledge that demonstrated the impact of the FWB intervention via different research methods; 4) the universality of the FWB approach which led to widespread application.ConclusionsThe FWB research impact exercise reinforced the view that assessing research impact is best approached as a “wicked problem” for which there are no easy fixes. It requires flexible, open-ended, collaborative learning-by-doing approaches to build the evidence base over time. Steps and approaches that research groups might take to build the research impact knowledge base within their disciplines are discussed.

The Marcus Institute for Brain Health: an integrated practice unit for the care of traumatic brain injury in military veterans

ABSTRACT Primary objective Traumatic brain injury (TBI) is a signature wound of recent Unites States military conflicts. The National Intrepid Center of Excellence (NICoE) has demonstrated that interdisciplinary care is effective for active-duty military personnel with TBI and related psychological health conditions. This paper details how the Marcus Institute for Brain Health (MIBH), established in 2017 as an Integrated Practice Unit (IPU), is founded on the NICoE model and is dedicated to interdisciplinary care for Veterans with persistent symptoms due to TBI and psychological comorbidities. Research design A highly integrated group of clinicians from diverse disciplines combine their expertise to offer comprehensive evaluation, intensive outpatient treatment, and program outcomes evaluation. Methods and procedures The role of each discipline in the provision of care, and the regular interaction of all clinicians, are delineated. A strong connection to academic medicine is maintained so that clinical research and education complement patient care. Main outcomes and results Over three hundred veterans and family members have received treatment at the MIBH. Program evaluation is underway. Conclusions As the understanding of TBI and related psychological conditions continues its rapid evolution, the expert interdisciplinary care at the MIBH has great promise as a Veteran counterpart of the NICoE

Commentary on Elbers et al. "Longitudinal outcome evaluations of IMPT programs".

"This journal recently published a paper by Elbers et al., entitled "Longitudinal outcome evaluations of interdisciplinary multimodal pain treatment programs for patients with chronic primary musculoskeletal pain: a systematic review and meta-analysis". It not only provides an evaluation of outcomes, but also a glimpse into the nature of these programs. Elbers and colleagues raise several issues of importance for clinicians, clinical researchers, and health service providers/funders.

Implementation and Outcomes of a Maternal Smoking Cessation Program for a Multi-ethnic Cohort in California, USA, 2012-2019.

Smoking during pregnancy remains one of the most significant risk factors for poor birth outcomes. During 2012–2019, the Loma Linda University Health Comprehensive Tobacco Treatment Program (CTTP) used a multicomponent behavioral intervention for tobacco cessation for 1402 pregnant smokers with components of known efficacy (i.e., incentives, biomarker testing, feedback, and motivational interviewing). The CTTP cohort includes a multi-ethnic sample of pregnant women with a mean age of 27 years referred by collaborating community-based healthcare providers in San Bernardino county. Evaluation of program outcomes from 7 years of follow-up (2012–2019) creates a rich cohort dataset for implementation science research to examine the real-world effectiveness of the program. In this report, we provide a cohort profile, and 8-week prolonged abstinence (8-week PA) and relapse findings from the first year of follow-up (n = 233). We found: (1) 28.4% achieved 8-week PA, (2) At a median of 6.2 months of follow-up after achieving 8-week PA, 23.2% of enrolled subjects reported tobacco cessation, and (3) a high rate of loss to follow-up (44%). In addition, our modeling indicated that the odds of relapse/smoking after enrollment was significantly higher in young mothers, non-Hispanic mothers (White, Black/African-American), mothers in the first and third trimester, and rural mothers. Formative quantitative and qualitative research on the CTTP cohort will consider the effects of a range of implementation science (number of intervention sessions, addition of a mHealth component, distance to care) and individual (partner/household smoking, birth outcomes, NICU) outcome measures for the purpose of scaling up the CTTP model.

Sustainability of Tobacco Treatment Programs in the Cancer Center Cessation Initiative.

The NCI's Cancer Center Cessation Initiative (C3I) has a specific objective of helping cancer centers develop and implement sustainable programs to routinely address tobacco cessation with patients. Sustaining tobacco treatment programs requires the maintenance of (1) core program components, (2) ongoing implementation strategies, and (3) program outcomes evaluation. NCI funding of C3I included a commitment of resources toward sustainability. This article presents case studies to illustrate key strategies in developing sustainability capacity across 4 C3I-funded sites. Case studies are organized according to the domains of sustainability capacity defined in the Clinical Sustainability Assessment Tool (CSAT). We also describe the C3I Sustainability Working Group agenda to make scientific and practical contributions in 3 areas: (1) demonstrating the value of tobacco use treatment in cancer care, (2) identifying implementation strategies to support sustainability, and (3) providing evidence to inform policy changes that support the prioritization and financing of tobacco use treatment. By advancing this agenda, the Sustainability Working Group can play an active role in advancing and disseminating knowledge for tobacco treatment program sustainability to assist cancer care organizations in addressing tobacco use by patients with cancer within and beyond C3I.

Evaluation of Programme Outcomes Under the Psychomotor and Affective Domain for Diploma Civil Engineering Students Through Industrial Training: A Statistical Study from Employers' Perspective in Malaysia

The psychomotor and affective are important domains in Programme Out-comes (PO) and are used to assess the achievement of learning objectives. The psychomotor domain refers to performing motor activities accurately, smoothly, and quickly. The affective components are those personality and behavioural characteristics which focuses on feelings, emotions, and attitudes. Through industrial training, students are exposed to engineering practices, the real civil engineering environments and the ethical and communication skill required. This paper identified the employer's perspective on the students’ attainment for psychomotor and affective domains while undergoing industrial training. There were 272 students are involved in this study that undergoing the Industrial Training in session II 2019/2020. The portion of marks from the industry evaluation for the psychomotor domain is 40% and for the af-fective domain is 10%. The marks were analyzed using descriptive statistical analysis to measure student’s performance. The Pearson correlation (r) was used to identify the relationship between each POs with related domain and the t-test analysis was completed to evaluate any significant difference be-tween the performance evaluation of male and female students. The result shows that most of the students were excellent in the psychomotor and affective domain. The r-value for each POs shows a moderately positive correlation. Therefore, the outcomes of this study can provide input to the faculty to improve the important elements of the psychomotor and affective domain to produce good quality of future civil engineering technical personnel.

A 1-year training program in emergency medicine for physicians in Karachi, Pakistan: Evaluation of learner and program outcomes.

Pakistan has an underdeveloped and overburdened emergency care system, with most emergency departments (EDs) staffed by physicians not formally trained in emergency medicine (EM). As of January 2020, only nine Pakistani institutions were providing formal EM specialty training; therefore, a training program of shorter duration is needed in the interim. The Certification Program in Emergency Medicine (CPEM) is a 1-year training program in EM consisting of two arms: CPEM-Clinical (CPEM-C), which includes physicians from The Indus Hospital (TIH) ED, and CPEM-Didactic (CPEM-D), including physicians from EDs across Karachi. Both groups participate in weekly conferences, including didactics, small-group discussions, workshops, and journal clubs. CPEM-C learners also receive clinical mentorship from visiting international and TIH EM faculty. Both groups were assessed with preprogram, midterm, and final examinations as well as on clinical skills. Additionally, both groups provided regular feedback on program content and administration. Twenty-five of the 32 initially enrolled learners completed the program in June 2019. Scores on a matched set of 50 questions administered in the pretest and final examination improved by an average of 15.1% (p<0.005) for CPEM-C learners and 8.5% (p<0.0005) for CPEM-D learners, with 93% of learners showing improvement. Clinical evaluations of CPEM-C and CPEM-D learners during the first and fourth quarters showed an average improvement of 1.1 out of 5 (p<0.05) and 1.2 out of 9 (p<0.0005) points, respectively. Learner evaluations of the program were overall positive. CPEM demonstrated significant improvement in test scores and clinical evaluations in both program arms. Evaluations also suggested that the program was well received. These data, along with CPEM's ability to train physicians from multiple institutions using low-cost, innovative educational strategies, suggest that it may be an effective, transferable mechanism for the expedited development of EM in Pakistan and countries where EM is developing as a specialty.

A systematic review of global surgery partnerships and a proposed framework for sustainability.

Background:Building surgical capacity through global surgery partnerships (GSPs) between high and low- and middle-income countries (LMICs) is a rising global health focus. Our aim was to conduct a systematic review to characterize strategies employed by GSPs to build capacity and promote sustainability and to propose a novel reproducible model for sustainability.Methods:We conducted a systematic review according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We searched PubMed, EMBASE, Medline and African Journals Online to identify all peer-reviewed articles published between 2000 and 2016 that described GSPs between partners from the United States or Canada or both and partners from LMICs. We excluded papers that described nonsurgical GSPs, unilateral GSPs (e.g., humanitarian missions) or military initiatives. Descriptive features were analyzed, with a focus on attributes that promote sustainability. We then proposed criteria for sustainability on the basis of the themes that emerged from our review.Results:Our search retrieved 3580 abstracts, which were then independently reviewed by 4 authors. A total of 128 papers (3.6%) met the inclusion criteria. They described GSPs in 68 countries on 5 continents. Among the GSPs, 21.9% demonstrated community engagement and 51.6% included multidisciplinary collaboration. Surgical training or education was provided in 81.3% of GSPs. Although 64.8% of GSPs collected data, only 53.1% reported project-related outcomes. A total of 55.5% had bilateral authorship for publications, and 28.9% had multisource funding. Only 1 GSP fulfilled all 6 of our criteria for sustainability.Conclusion:In this systematic review we identified 6 pillars that are indicators of sustainability: community engagement, multidisciplinary collaboration, education and training, outcomes reporting, bilateral authorship and multisource funding. We propose that future GSPs should build on a foundation of bilateral ideas and expertise exchange, that they should have defined and measurable objectives, that they should engage in continuous evaluation of program outcomes and that they should take a thoughtful and transparent approach to sustained capacity building.

130 Comparison of program evaluations for an extension short course offered in-person and virtually

The COVID pandemic forced many traditional in-person extension programs to become virtual. The Equine Environmental Stewardship Short Course (EES) focuses on pasture and manure management and their impacts on surface water quality. In 2020, EES was converted to a virtual format due to pandemic restrictions. The objective of this project was to compare the program evaluation results of the EES short course when offered virtually versus in-person. The EES consists of 4 2-h evening sessions (2017 in-person, 2020 via Zoom). Handouts were given in class in 2017 and mailed in 2020. Information was presented via PowerPoint on 10 different topics. Discussions were facilitated in 2020 using breakout rooms and in 2017 with normal event breaks. Evaluations were collected after each night (2017 via paper, 2020 via Qualtrics). Data were collected on demographics, knowledge increases (1–4 scale with 4 being high), and intention to complete management practices. Welch's t -test and Fischer's exact test were used to analyze data, with significance set at P < 0.05. The 2017 EES had 25 people register (from 1 state), while 29 registered (from 6 states) for 2020. Survey response rate varied by night (2017: n = 17–22; 2020: n = 6–11). Demographics between the years were similar ( P > 0.05). Participants kept an average of 6.4 horses on 10.7 acres and rated the quality of their pastures as 2.8 on a scale of 1 to 5 (5 as highest quality). The overall mean knowledge increase was higher in 2020 (3.4 vs 2.9; P < 0.001), with 2020 participants reporting higher knowledge gains on forage biology, resting pastures, weed life cycles, soil test interpretation, methods for pasture renovation, and ideal manure storage features ( P < 0.05). Mowing pastures was the only adopted practice that differed ( P < 0.02). On night 1 of 2020 EES, 18% of participants had technical difficulties, but that dropped to 0–9% for subsequent nights. Breakout rooms were used 2 nights in 2020, and 90% and 67% of participants liked them on nights 1 and 2, respectively. Improved knowledge scores in 2020 could be due to slightly shortened talks (same content as 2017) and some different presenters. Overall practice adoption plans did not differ, possibly because each cohort reported similar quality pastures and thus had similar goals for improvement. In 2020, most participants reported few technology issues, but virtual challenges included lower evaluation response rate and some registrants opting to view recordings instead of participating live. Offering an in-person extension program as a virtual program is a viable option with the potential to reach a geographically wider audience.

Development and validation of a Japanese version of the TRANSITION-Q.

The evaluation of transition readiness is indispensable for long-term follow-ups of adolescent patients with childhood-onset chronic diseases (CCD). We developed a Japanese version of the TRANSITION-Q (TRANSITION-Q-J) and used it to assess Japanese patients with CCD. The TRANSITION-Q-J was developed through forward and backward translations followed by cognitive interviews with five adolescent patients. The field test was conducted with 125 adolescent patients, and a retest was conducted with 113 adolescent patients. Confirmatory factor analysis supported the two-factor analysis model including F1 (communication and self-management) and F2 (examination behavior). Sufficient internal consistency and test-retest reliability were demonstrated among the total 14 items, F1, and F2 (Cronbach's α>0.80, intraclass correlation coefficient>0.85). Convergent and discriminant validity for the 14 items and F1 were acceptable; however, F2 did not correlate significantly with the Rosenberg Self-Esteem Scale and Independent Consciousness Scale. Regarding known-groups validity, the older group had a significantly higher mean TRANSITION-Q-J score (50.05) than the younger group (43.28; P=0.04). The same results were found for both F1 and F2. The TRANSITION-Q-J for adolescent patients with CCD was developed and its reliability and validity were verified. This scale is easy to administer. In addition to being a tool for transition period support, it could be used to verify effective factors and in program outcome evaluation, including intervention studies.