Program Evaluation Methods Research Articles

Qualitative evaluation of a multidisciplinary master of cancer sciences: impacts on graduates and influencing curricular factors

BackgroundEvaluations of continuing professional development programs typically focus on short-term knowledge and skill acquisition. There is a need for more comprehensive program evaluation methods that assess a broader range of impacts and can elicit how and why these outcomes occurred. We conducted a qualitative study to investigate the impacts of a multidisciplinary, online health professional postgraduate degree and to gain insights into the factors that led to these impacts.MethodsParticipants were graduates of the University of Melbourne’s Master of Cancer Sciences who could participate in an online interview. Semi-structured, qualitative interviews were conducted exploring a broad range of impacts, including changes in professional practice and career trajectory since graduation, and how the degree influenced these impacts. Data were analysed inductively.ResultsFifteen participants (female: 80%, 31–50 years old: 67%) from a range of professions were interviewed. A number of major themes were uncovered. Impacts on career trajectory included expanded career horizons (e.g. increased role diversity and complexity), and increased confidence in their professional identity. Impacts on professional practice included individual improvements in patient care and research, as well as changes in organisational practice. Factors identified as leading to these impacts were: (i) active, interactive and interprofessional learning; (ii) networking, informal mentoring, and role-modelling; and (iii) support at multiple levels.ConclusionThis study provides preliminary evidence of the positive impact of a Master of Cancer Sciences on graduate career trajectory and professional practice. In addition, the inductive methodology enabled identification of the curricular features (both planned and emergent) that influenced these impacts, facilitating potential transferability of learnings to other teaching programs.

National mixed methods evaluation of Adult and Youth Mental Health First Aid

ObjectiveMental Health First Aid (MHFA) is an international, early intervention, public education program that teaches participants to identify, understand, and respond to signs of mental health and substance use challenges. Until now, all evaluations on MHFA courses in the United States have been performed externally, which have limits in generalizability and the ability to address insights in a timely manner. Additionally, few evaluations have assessed the newest versions of MHFA courses. Therefore, this evaluation aimed to understand key outcomes and participant responses to Adult and Youth MHFA courses in a nationwide sample, spanning nearly a year of real-world implementation. MethodsAn internal, mixed methods program evaluation was performed on the second edition of the in-person versions of Adult and Youth MHFA. Pre- and post-course evaluation data from 3,586 Adult and 2,314 Youth MHFA participants trained between October 2022 and July 2023 were used in this evaluation. Key quantitative constructs, such as mental health literacy, intentions to perform MHFA skills, and confidence to use MHFA skills, were analyzed using paired sample t tests. Rapid thematic analysis methods were used to analyze participants' qualitative responses to the course. ResultsSignificant increases in mental health literacy, intentions, and confidence were observed in both Adult and Youth MHFA participants from pre- to post-course. Qualitative analyses revealed key themes in participant responses, including the helpfulness of the activities, materials, and the MHFA action plan. Adjustments to materials, the length or pace of the course, and other specifics were noted as opportunities for improvement. These findings will be used to inform the future practice, implementation, and revisions of MHFA. ConclusionCompletion of Adult or Youth MHFA by adults in the United States led to improvements in mental health literacy and intentions and confidence to assist someone in need. Both programs may benefit from implementation and content-related changes to enhance participant experience.

Realist process evaluation of the knowledge translation programme of a patient‐oriented research network

AbstractRationaleThe Knowledge Translation (KT) Programme of a pan‐Canadian strategic patient‐oriented research network focused on brain‐based developmental disabilities aimed to mobilize knowledge relevant to the network members. The programme also promotes and studies integrated Knowledge Translation (iKT) approaches involving different interested parties, such as researchers, patient‐partners and decision‐makers, in all parts of the knowledge creation process.Aims and ObjectivesThe objective of this study is to advance research programme evaluation methods through a realist evaluation of the process of implementing iKT activities.MethodsRealist process evaluation included: (1) development of initial programme theories (using the partnership synergy theory); (2) data collection and analysis; (3) synthesis and refinement of theories through engagement with literature; and (4) presentation of findings in context‐mechanism‐outcome (C‐M‐O) configurations. A range of project documentation records were reviewed for analysis, and three co‐leads, a programme coordinator, and a senior research associate were consulted to contextualize the implementation process of relevant KT activities.ResultsBased on the developed C‐M‐O configurations, we identified five key mechanisms of generating synergy in the iKT processes: (1) Visible shared leadership that embodies what iKT looks like; (2) Researchers' readiness for iKT; (3) Adaptation and flexible allocation of resources to emerging needs; (4) Power sharing to create practical and creative knowledge; and (5) Collective voice for potential transformative impacts at the policy level.ConclusionsThe current realist evaluation demonstrated how partnerships between researchers, patient‐partners and other interested parties can synergistically generate new ways of thinking among all interested parties, actionable strategies to integrate users in research, and solutions to disseminate knowledge. In particular, we identified a pivotal role for patient‐partners to act as equal decision‐maker helps building and maintaining partnerships and consolidating KT strategies.

Experiences and perceived benefits of remotely delivered dance/movement therapy for adult cancer patients: a multi-method program evaluation.

Dance/movement therapy (DMT) is increasingly used as a complementary treatment to address psychological and physical wellbeing. However, it is unknown how it can be leveraged in adult cancer care. This mixed methods program evaluation aimed to assess patient-reported benefits and satisfaction with the virtual DMT in an academic oncology setting. We developed, implemented, and evaluated a 6-week virtual, synchronous DMT program aiming to improve physical health, address mental distress, and foster social connection for cancer patients. We used deidentified program evaluation data to assess impact of DMT on patient-reported outcomes and patients' satisfaction with the DMT program. Pre- and post-session data were analyzed using descriptive statistics and a paired t-test. Qualitative data were captured through semi-structured interviews and analyzed using thematic analysis. Results from 39 participants (mean age 64.7 ± 9.8), majority female (89.7%) with a history of breast cancer (43.6%), showed high satisfaction (100%) and unanimous program recommendation (100%). Significant improvements were noted in anxiety (- 0.42 ± 0.76, p = .009), distress (- 0.35 ± 0.80, p = .036), and sense of joy (0.73 ± 1.18, p = .004), with a non-significant trend in increased physical activity (0.38 ± 0.98, p = .057). Thematic findings indicated that DMT participation (1) facilitated engagement in physical activity for improved physical health, (2) fostered creative expression, (3) improved mental state, and (4) helped build social connections and support. Our DMT program shows promise as a component of integrative cancer care. The mixed-method evaluation provides insightful information to generate hypotheses for future RCT studies aiming to evaluate the specific effects of DMT on patient experience and outcomes.

A Guiding Model for Undergraduate Medical Education Well-Being Programs.

Most medical schools have instituted undergraduate medical education (UME) well-being programs in recent years in response to high rates of medical student distress, but there is currently significant variability in the structure of UME well-being programs and limited guidance on how to best structure such programs to achieve success. In this article, the authors, all leaders of medical student well-being programs at their home institutions and members of the Association of American Medical Colleges Group on Student Affairs Committee on Student Affairs Working Group on Medical Student Well-Being between 2019 and 2023 offer guidance to the national community on how best to structure a UME well-being program. They use the current literature and their professional experiences leading well-being efforts at 7 different institutions to review the case for addressing medical student well-being, propose a guiding model, and make recommendations for strategies to implement this model.The proposed guiding model emphasizes the importance of the learning environment and efficiency of learning to medical student well-being, as well as personal resilience. Based on this model, the authors recommend specific and tangible well-being strategies to implement systemic interventions to improve the learning environment, efficiency of learning, and personal resilience, including formalizing the well-being program; hiring qualified, dedicated, and empowered well-being leadership with clear responsibilities; acting as a central hub for resources and as a liaison with mental health care; and establishing robust program evaluation methods.

Physician engagement in reproductive health advocacy: findings from a mixed methods evaluation of a leadership and advocacy program

BackgroundMedical curricula include advocacy competencies, but how much physicians engage in advocacy and what enables this engagement is not well characterized. The authors assessed facilitators and barriers to advocacy identified by physician alumni of a reproductive health advocacy training program.MethodsThe authors present secondary results from a mixed methods program evaluation from 2018 to 2020, using alumni data from a cross-sectional survey (n = 231) and in-depth interviews (IDIs, n = 36). The survey measured engagement in policy, media, professional organization, and medical education advocacy and the value placed on the community fostered by the program (eight questions, Cronbach’s alpha = 0.81). The authors estimated the association of community value score with advocacy engagement using multivariable Poisson regression to estimate prevalence ratios and analyzed IDI data inductively.ResultsOver one third of alumni were highly engaged in legislative policy (n = 90, 39%), professional organizations (n = 98, 42%), or medical education (n = 89, 39%), with fewer highly active in media-based advocacy (n = 54, 23%) in the year prior to the survey. Survey and IDI data demonstrated that passion, sense of urgency, confidence in skills, and the program’s emphasis on different forms of advocacy facilitated engagement in advocacy, while insufficient time, safety concerns, and sense of effort redundancies were barriers. The program community was also an important facilitator, especially for “out loud” efforts and for those working in environments perceived as hostile to abortion care (e.g., alumni in hostile environments with high community value scores were 1.8 times [95% CI 1.3, 2.6] as likely to report medium/high levels of media advocacy compared to those with low scores after adjusting for age, gender, and clinical specialty).ConclusionPhysician advocacy training curricula should include both skills- and community-building and identify a full range of forms of advocacy. Community-building is especially important for physician advocacy for reproductive health services such as abortion care.

Program Evaluation and Improvement by a Homeless Community Using a Human Centered Design Approach

People experiencing homelessness are often absent from the development and implementation of new housing projects. As mounting evidence demonstrates the link between health and housing, it seems ineffective to design solutions without the participation and insight of those who are chronically unhoused. This study describes a method for program and product evaluation that people with lived experience of homelessness can use to determine the value of new offerings and then design improvements based on their evaluation. Specifically, this article reports the experience of evaluating a handbook describing the operations of a new type of housing for people experiencing homelessness. The research team, which included people with lived experience of homelessness, performed interviews to evaluate plans in the handbook, analyzed those results, and designed improvements in the operations of the housing based on the program evaluation. The team utilized participatory analysis of the interview data and then used a Human-Centered Design approach to generate ideas based on the analysis and develop concepts to improve the operational plan. The evaluation identified four high-priority problems with the operational plan and the group then designed three unique solutions to address those problems. This project demonstrates that people with lived experience of homelessness can participate as equal partners in the evaluation of a program and the subsequent program improvement design that was based on their evaluations.

185 A Clinical and Translational Science manuscript writing support program for research staff

OBJECTIVES/GOALS: The objective of this initiative was to promote MICHR staff’s production of Clinical and Translational Science publications. MICHR leadership approved this initiative, including an evaluation plan with measurable outcomes goals, and contracted with an experienced scientific writing coach with over 20 years of experience working with CTSAs. METHODS/STUDY POPULATION: A sequential mixed methods program evaluation designs was used. Pre- and post-surveys were used to measure participating staff’s gain in skill, understanding & satisfaction. An interview with the instructor was then conducted to characterize staff performance, and identify possible areas of programmatic improvement. This initial phase of the program evaluation was conducted in the Summer of 2023. The results were used to inform an expansion of the program to include more staff in the the Fall of 2023. Pre- and post-program surveys of the participants were conducted and interviews with each program participant were conducted. Finally, interviews with non-participating staff were conducted to assess their need for writing support and the challenges and facilitators of their scientific writing. RESULTS/ANTICIPATED RESULTS: Preliminary evaluation results obtained in the summer of 2023 were positive. All participants completed the course, and spent an average of 3.3 hours working between sessions. Six manuscripts were developed, five of which are being readied for submission and one submitted as of August 2023. Analysis of the pre-and post-program surveys indicated that all participants gained writing skill and authorship knowledge. Specifically, they gained confidence in 7 writing skills and 6 first-author roles evaluated during the course. All participants were satisfied with their experience and recommended the course to their colleagues, and the course instructor was also satisfied with the course. However, the participants noted that competing work demands and variable preparation hindered their work in the course. DISCUSSION/SIGNIFICANCE: With the increasing focus on Clinical and Translational Science taking place across the CTSA Consortium it is important to involve research staff in paper writing teams, including in first-author roles. Professional development in scientific writing can support Clinical and Translational Research staff contributing to this emerging science.

Evaluating Pre-trial Programs Using Interpretable Machine Learning Matching Algorithms for Causal Inference

After a person is arrested and charged with a crime, they may be released on bail and required to participate in a community supervision program while awaiting trial. These 'pre-trial programs' are common throughout the United States, but very little research has demonstrated their effectiveness. Researchers have emphasized the need for more rigorous program evaluation methods, which we introduce in this article. We describe a program evaluation pipeline that uses recent interpretable machine learning techniques for observational causal inference, and demonstrate these techniques in a study of a pre-trial program in Durham, North Carolina. Our findings show no evidence that the program either significantly increased or decreased the probability of new criminal charges. If these findings replicate, the criminal-legal system needs to either improve pre-trial programs or consider alternatives to them. The simplest option is to release low-risk individuals back into the community without subjecting them to any restrictions or conditions. Another option is to assign individuals to pre-trial programs that incentivize pro-social behavior. We believe that the techniques introduced here can provide researchers the rigorous tools they need to evaluate these programs.

Integration of Total Worker Health ® Training for Occupational Health and Safety Professionals With Efforts to Promote Worker Health, Safety, and Well-being in Mexico.

We sought to test whether a 2-week Total Worker Health (TWH) training mapped to TWH education competencies could be administered to a Mexican audience of occupational safety and health professionals and could lead to positive changes to knowledge and behaviors. This study used robust program evaluation methods collected before and after each of the nine training days and at the end of the course. Overall course quality received a mean score of 4.6 (SD = 0.6) and 98.8% of participants agreed that their TWH knowledge increased. All participants intended to make at least one change to their professional practice, most frequently helping companies assess their organizational culture to support health, safety, and well-being. This TWH training was well received and led to positive self-reported increase in knowledge and abilities to influence workers' health, safety, and well-being.

Schedule risk model of water intake tunnel construction considering mood factors and its application

The mood index r\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$r$$\\end{document} was used to describe evaluator attitudes regarding the progress of a project that formed the basis of a construction period prediction model. The degrees of pessimism α\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\alpha$$\\end{document} and optimism β\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\beta$$\\end{document} were introduced, and an analysis model was established using α\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\alpha$$\\end{document} and β\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\beta$$\\end{document} to predict the construction period and completion probability Firstly, the absolute construction period of each process of tunnel No. 2 can be obtained according to the measured daily average footage of each process of tunnel No. 1. Secondly, the probability of the stoppage caused by different factors can be obtained after the statistical analysis of the factors responsible for the stoppage of tunnel No. 1. Finally, the expected construction period and completion probability of tunnel No. 2 under different pessimism and optimism conditions are obtained by using the progress risk analysis theory of emotional models and the program evaluation and review technique method. An engineering application showed that the expected construction period increased, and the completion probability decreased considerably with increasing pessimism; the opposite trend occurred as optimism increased. During the process of risk management and control, the prediction model can be used to perform precise quantitative analysis of the expected construction period and completion probability, reduce the blindness of construction management, control decisions of complex giant tunnel projects, and provide a more accurate basis for decision makers to judge risks. The findings of this study can be applied to hydraulic tunnels and can provide a reference for traffic tunnels, railway tunnels, and other similar projects.

Navigating the Path to Recovery: A Scoping Review of Return to Work Program for Manufacture Workers After Occupational Injury

Introduction: Workplace accidents in the manufacturing industry may significantly affect both a worker's capacity to return to work and their general well-being. Effective return-to-work programs may help disabled workers rehabilitate. The present scoping review endeavors to scrutinize the extant body of literature pertaining to Return to Work programs tailored specifically for individuals employed in the manufacturing industry who have experienced occupational injuries. Methods: A comprehensive exploration was conducted across various scholarly databases, including PubMed, Scopus, and Web of Science. The search technique includes return-to-work, occupational injury, and manufacturing keywords. Inclusion criteria were used to screen studies. Results: A total of 5 studies were included in the final analysis, representing a range of program types, implementation strategies, and evaluation methods. The results of this review suggest that return-to-work programs can be effective in facilitating workers' return to work after occupational injury and promoting positive health outcomes. Key program components identified as contributing to program effectiveness included workplace accommodations, early return-to-work programs, and rehabilitation services. Conclusion: This scoping review emphasizes the necessity of return-to-work programs for wounded manufacturing employees. The findings of this comprehensive review indicate that return-to-work programs when meticulously crafted and effectively executed, possess the capacity to assume a pivotal function in facilitating the reintegration of injured workers into the workforce, while concurrently enhancing their holistic health outcomes. Additional investigation is warranted to enhance comprehension of the precise constituents of a program that contribute to its triumph and to delve into the efficacy of diverse implementation methodologies.

Benefits of Kalteng Berkah Scholarship, What's for and to Whom? (Evaluation of Scholarship Program Using a Transactional Model)

Background: Based on the 2016-2021 Central Kalimantan Long Term Development Plan (RPJMD) contained in Regional Regulation Number 1 of 2017 (2), the vision of the Central Kalimantan provincial government is called Kalteng Berkah. "Kalteng Berkah" is an abbreviation for Central Kalimantan, which means dignified, graceful, religious, strong, fair, and harmonious. The Kalteng Berkah Program focuses on nine missions, including accelerating infrastructure development, increasing regional income, controlling inflation, alleviating poverty, and improving the quality of education and health. This mission gave birth to several programs, including the Central Kalimantan Berkah program in the field of education. Aim: Based on the technical guidelines for the Bidikmisi Kalteng Berkah program, this program aims to help the people of Central Kalimantan who need funding for their studies or to complete studies at universities. Method: This study used a transactional model program evaluation method. The paradigm was a mixed qualitative-quantitative paradigm. A qualitative approach was used to analyze the awardee's responses descriptively. Results and Dıscussion: the program has achieved the expected goals. In general, achieving objectives as perceived by the awardee indicates the program's success and may be used as consideration for policymaking to continue the program. Conclusion: By using the transactional evaluation model, we concluded that the Kalteng Berkah Scholarship program has been successfully implemented following the expected program objectives.

Measuring the development of a medical professional identity through medical school

Purpose The Professional Identity Essay (PIE) is a theory and evidence-based Medical Professional Identity Formation (MPIF) measure. We describe trajectories of PIE-measured MPIF over a 4-year US medical school curriculum. Methods Students write PIEs at medical school orientation, clinical clerkships orientation, and post-advanced (near graduation) clerkship. A trained evaluator assigns an overall stage score to narrative responses to nine PIE prompts (inter-rater ICC 0.83, 95% CI [0.57 − 0.96], intra-rater ICC 0.85). Distribution of PIE stage scores across time points were analyzed in the aggregate and individual students were classified as Increase, Stable (no score change) or Decrease based on the trajectories of PIE stage scores over time. Results 202 students completed 592 PIEs from 2018-2023. There was a significant change in the proportion of PIEs in stages over time (X2 84.40, p < 0.001), 47% (n = 95) students were categorized in the Increase trajectory, 45.5% (n = 92) as Stable and 7.4% (n = 15) as Decrease. Older age and time-predicted stage scores change within trajectories (p < 0.05). Conclusions Medical students’ PIE stage scores increase over time with three distinctive trajectories. Further study is needed to explore the utility of this method for formative assessment, program evaluation, and MPIF research.

Mastering Construction Schedules: The Power of CPM and Pert Integration

Abstract: Mastering construction schedules is a crucial task in project management, requiring a careful strategy for timely completion and overall success. This overview explores how Critical Path Method (CPM) and Program Evaluation and Review Technique (PERT) methods work together, highlighting the strength that comes from combining them. CPM, a certain method, focuses on putting tasks in order to find the critical path, which decides the project's duration. PERT, a method that considers uncertainties, provides a realistic view of timelines. Blending these methods creates a strong scheduling framework that balances certain and uncertain elements. This hybrid approach identifies critical paths, offering project managers a flexible scheduling model that adapts to uncertainties, helping them make more informed decisions. The abstract emphasizes the role of technology in integrating CPM and PERT, using computational power to seamlessly bring them together. Modern project management software automates critical path analysis, allowing real-time adjustments based on project conditions. This technology-driven integration is vital in today's project management. Moreover, the abstract highlights that CPM and PERT integration goes beyond traditional construction management, making it versatile for uncertain and dynamic projects, including research, development initiatives, and innovative construction. The adaptability of this integrated approach makes it valuable in various industries, indicating its broader impact on project scheduling. This overview outlines the core principles of the integration, demonstrating its practical use in addressing construction project management challenges. Embracing this integrated approach empowers project managers to navigate uncertainties, optimize resource use, and increase the chances of project success in a complex and dynamic environment.

Short-Term, Community-Based, Slow-Stream Rehabilitation Program for Older Adults Transitioning from Hospital to Home: A Mixed Methods Program Evaluation.

Shortened hospital stays have shifted the burden of care for older adults to community, informal (ie, family, caregiver) and formal post-acute care and services, highlighting the need for effective post-hospital stay services and programs. As there is a dearth of information related to community-based, slow-stream rehabilitation program models for older adults transitioning from hospital to home in the Canadian context, the paper describes a mixed methods evaluation of such a program. A mixed methods program evaluation, with process- and outcome-related elements, included 1) review and analysis of program documents; 2) observations to examine fidelity. Observation data were coded and summarized using descriptive statistics. Coded information and data were compared to document review data; 3) quantitative assessment of pre-post changes in physical, social, and psychological outcome measure and instrument scores using descriptive statistics, paired t-tests and confidence intervals (p = 0.05); and 4) exploration of acceptability through interviews and focus groups with 41 of the older adult participants and 17 family caregivers. Thematic analysis was used to examine focus group and interview transcripts. Observational data indicated alignment with the program document information overall. Statistically and clinically significant positive trends in improvement for physical outcome measure scores were observed (6-minute Walk Test, Life Space Assessment, Short Physical Performance Battery, Rapid Assessment of Physical Activity). Participants and family caregivers identified several positives and benefits of the program, ie, improvement in physical, social and mental well-being, decreased caregiver burden; and areas for improvement ie, need for more information about the program prior to enrollment and individualization, several of which aligned with the observation and quantitative data. This mixed methods program evaluation provided a detailed description of a community-based, slow-stream rehabilitation program for older adults who are transitioning to home post-hospital stay and its participants. Evidence of program fidelity, acceptability, and positive trends in improvement in physical outcome measure scores were found. Information about program strengths and areas for improvement can be used by stakeholders to inform program refinement and enhancement.

대학의 일통합 학습모형 적용에 관한 탐색적 연구

This study was conducted to develop a work-integrated learning model to intentionally link and integrate students' work experiences in future job sites and university curriculum, and to propose application plans focusing on real estate and tourism majors. To achieve this research purpose, theoretical and practical prior studies and domestic and foreign cases of work-integrated learning were reviewed, and based on this, a work-integrated learning model consisting of four types: a full-course integrated model, a specific area advanced model, a specific topic-oriented model, and a multi-process diversification model was proposed. Due to the nature of the major as a specific integrated learning application plan, the real estate major was presented as a specific topic-oriented model, and the tourism major was presented as a full-course integrated model. The implication of this study is that it presented a work-integrated learning model that can be implemented in universities and specific application plans centered on real estate and tourism majors. The limitation of this study is that there is no related research in Korea, so it is presented as an exploratory study, and future studies will require research on effective detailed program application and evaluation methods for each integrated learning model.

Public Libraries Can Be Open Science Laboratories for Citizen Science Projects

A Review of: Cigarini, A., Bonhoure, I., Vicens, J., &amp; Perelló, J. (2021). Public libraries embrace citizen science: Strengths and challenges. Library &amp; Information Science Research, 43(2), 101090. https://doi.org/10.1016/j.lisr.2021.101090Objective – The objective of this study was to evaluate the potential of libraries supporting citizen scientist (CS) projects. Design – Mixed methods program evaluation study. Setting – 24 public libraries in Barcelona, Spain. Subjects – Public librarians and library users. Methods – It is a mixed methods and mixed population study done in several phases. The first phase involved training 30 librarians how to conduct a citizen science project. They were given a pre and post survey about their perceptions of citizen science and comfort-level in conducting a project. The second phase involved a project run by the now-trained librarians with library user participation. At this phase a questionnaire was given to the users at the start and end of the project. Finally, a focus group of librarians was asked about their project. The responses were evaluated through thematic analysis. Seven libraries participated in the focus groups. Main Results – During the first phase of the study, the survey found the librarians were pessimistic about user participation in a citizen science project, both at the beginning (75%) and at the end (79%) of the session. Though they felt confident in discussing citizen science (100%) and had high satisfaction in the training (70%), only 42% felt confident to conduct a project on their own. The second phase involved the users, 94% of whom had never participated in a CS project. At the end, 70% of users said the project positively changed their perceptions of the library and 70% were satisfied with the experiment. During the focus groups, librarians said the project brought new users into the library and had the potential to build more relationships among participants and with the community. Major challenges discussed were user commitment to the project and the workload required by librarians, however they all answered positively when asked about continuing with CS projects. Conclusion – This study showed that citizen science projects can be successfully implemented in public libraries. Public libraries are facing challenges caused by societal change, the rise of open science, and more transparent and novel democratic ways of knowledge production. Updating public library infrastructure would be needed to support these projects more fully. This may involve building partnerships and developing new guidelines. There is potential for public libraries to be leaders and innovators in citizen science.

Use of Continuous and Complementary Program Evaluation Methods to Inform the Approaches of Rethink Your Drink Nevada: a Community-based Program Developed to Reduce Children’s Intake of Sugar-sweetened Beverages

Use of Continuous and Complementary Program Evaluation Methods to Inform the Approaches of Rethink Your Drink Nevada: a Community-based Program Developed to Reduce Children’s Intake of Sugar-sweetened Beverages

Evaluation of the Stunting Program at the Margototo Health Center, Lampung East District

Stunting is a condition of failure to thrive in children under five due to chronic malnutrition, especially in the first 1,000 days of life. Stunting can affect brain growth and development. Children with stunting have a greater risk of suffering from chronic diseases in adulthood. This study aims to analyze the evaluation of the implementation of the stunting program at the Margototo Health Center, East Lampung Regency. This research is a program evaluation research using a mixed method (Mix Method) with a sequential explanatory design where quantitative data analysis is carried out at the initial stage, then qualitative data analysis is carried out. The program evaluation method uses the context-input-process-product (CIPP) evaluation model. Results study shows that in terms of context, the stunting program has been implemented according to the planned objectives and is right on target. In the input aspect, namely the lack of human resources in assisting, the lack of a budget that supports program implementation. In the process aspect of program implementation, there were no significant obstacles, and the process of implementing the stunting program went according to plan. In the product/output aspect, the stunting program has reached the expected target indicators, and recording and reporting have been going well. And the results of the implementation of the stunting program are shown by the results of mothers' knowledge about stunting and the attitudes of mothers who have stunted toddlers towards stunting events.