Kirkpatrick's Evaluation Model Research Articles

Evaluating the Effectiveness of Existing In-service Training Courses on Infection Prevention and Control in Nurses: An Evaluation Using the Kirkpatrick Model

Background: Evaluating the effectiveness of educational courses is essential for ensuring high-quality healthcare. Objectives: This study assessed the effectiveness of current in-service training courses on infection prevention and control for nurses, using the Kirkpatrick evaluation model. Methods: This evaluative study was conducted at Shiraz Army Hospital in 2024, involving 40 nurses and 10 supervisors. The educational program consisted of interactive workshops held over two days, covering topics such as standard precautions, isolation procedures, and hand hygiene. The evaluation followed Kirkpatrick’s four levels: Reaction, learning, behavior, and impact. Nurses' reactions and knowledge were measured using validated questionnaires, while supervisors assessed behavioral changes. Data were analyzed with SPSS 26, using descriptive statistics and a one-sample t-test. Results: All four levels of the Kirkpatrick model were evaluated. Nurses reported a mean reaction score of 3.73 (SD = 0.80), reflecting positive feedback on the training. Knowledge scores significantly increased from a pre-test mean of 2.39 (SD = 0.74) to a post-test mean of 3.72 (SD = 0.74) (P < 0.001). Supervisors observed a behavioral improvement, with scores increasing from 2.34 (SD = 0.94) to 3.72 (SD = 0.74) (P = 0.004). Furthermore, the nosocomial infection index decreased from 0.7 to 0.5 (P = 0.002) following the training. Conclusions: The findings demonstrate the effectiveness of current in-service training courses on infection prevention and control for nurses. The Kirkpatrick model proved to be a valuable evaluation tool, underscoring the importance of ongoing assessment of nurses’ competencies to enhance infection prevention practices.

Advancing police-researcher collaboration and evidence-based policing: an evaluation of the applied criminology and data management course

ABSTRACT While the demand for evidence-based policing (EBP) has increased and researchers continue to generate important research findings, few targeted educational efforts have been made to nurture effective police-researcher collaboration and integrate EBP into police practice. The current study evaluates a 16-month-long Applied Criminology and Data Management Course (ACDC), which aimed to train police executives and early-career researchers to work together, produce EBP research, and disseminate their findings. Survey and observational data assess participant progress on four key Kirkpatrick evaluation model metrics: reaction, learning, behavior, and results. ACDC participants demonstrated satisfaction with the program and knowledge of skills required for effective collaboration and research. Most notably, they demonstrated application of skills through applied research and dissemination of results. Participants proved committed to advancing EBP via trainings to police executives and community stakeholders, publications, presentations, promotions to various relevant positions, and secured grant funding. Overall, these preliminary results suggest that ACDC can serve as a successful and feasible model for advancing EBP.

Preparing health students for climate change: the impact of an innovative ‘Plan Health Faire’ course

Abstract Introduction Recent studies highlighted the need for climate change (CC) training in universities, especially for health students. In France, a collaborative teaching program, “Plan Health Faire” (PHF), addresses this need. This study aimed to assess PHF’s impact on health students’ reactions, knowledge, and behaviour. Methods A cross-sectional study was carried out among all second-year health students enrolled in Medicine, Pharmacy, Odontology, and Maieutics at the University of Poitiers in September 2022. During a mandatory seminar, all students took part in the innovative and scientific PHF teaching program which aims to raise awareness of the challenges of CC as applied to healthcare. Two structured questionnaires based on the Kirkpatrick evaluation model were administered before and after the course. Using a comparative statistical analysis, the changes in students’ knowledge were assessed. Results In all, 294 students took part in the study, and 97.4% were satisfied or very satisfied with PHF. An average score of 8.6/10 was given to the collaborative dynamics of the training. In terms of changes in their level of knowledge and their perception of these changes, 98.3% of students felt that they had learned new knowledge about CC and healthcare. A comparative analysis of the answers to the knowledge questions showed that students’ level of knowledge is indeed significantly higher at the end of the course (p<0.0001). Finally, 94% of students expressed their probable or certain intention to integrate these issues into their future professional practice. Conclusions These results indicate that the PHF training program has a significant impact on health students, particularly in terms of satisfaction and knowledge. PHF could be an interesting way to teach health students about the challenges of CC in healthcare. PHF is currently tested in other French Universities and health care facilities. Key messages • Plan Health Faire is an innovative and collaborative educational program aimed at raising awareness among healthcare stakeholders about the challenges of climate change. • Plan Health Faire is an effective program for raising awareness among health students about climate change and is also an interesting lever for integrating climate change into their future practice.

Serratus Anterior Plane Block Remote Learning Curriculum.

Regional anesthesia aids in management of acute pain in the emergency department, but many emergency physicians remain inadequately trained. Further complicating medical education, our academic center continues to use remote learning as the primary setting for residency didactics. This project aims to create a remote conference session on ultrasound-guided serratus anterior plane blocks (USG-SAPB). We used the ADDIE (analyze, design, develop, implement, evaluate) model for curricular design, with emergency medicine residents as our intended learners. For the analyze element, we examined clinical need and resident program evaluation feedback. For design, we utilized best practices for remote learning, principles of mental rehearsal, and multimodal instructional theory. For develop, we completed recruitment of faculty leads, materials, and beta testing of each component. We implemented our 50-minute session on the videoconferencing platform Zoom. For evaluate, we created a program evaluation survey based on Kirkpatrick's evaluation model. Seventeen learners completed the evaluation. For Kirkpatrick level 1, 94% reported being very or extremely satisfied. For Kirkpatrick level 2, 91% ranked their presession confidence level in performing USG-SAPB as not at all or slightly confident, compared to the postsession, where 88% ranked themselves as moderately, very, or extremely confident. For Kirkpatrick level 3, 94% reported that they would incorporate at least a moderate amount of content into clinical practice. This remote learning session on USG-SAPB demonstrated high levels of learner satisfaction, improvement in learner confidence, and potential to impact learners' clinical practice. Future studies can elucidate patient outcomes related to educational sessions.

The development and evaluation of the Designing for Dissemination and Implementation Learning Hub.

There is a growing emphasis on reducing the gap between research and routine practice. Dissemination and Implementation (D&I) science offers theories, models, and frameworks to enhance the implementation, impact, and sustainment of new programs and interventions. Few training opportunities are available that help leaders, researchers, clinicians, and staff (implementers) translate original research into practice settings in a more timely and effective manner without requiring significant time away from their primary clinical duties. To address these needs, we designed a virtual Designing for Dissemination and Implementation (D4D&I) Learning Hub that offered foundational D&I knowledge and opportunities for skill building. We developed the D4D&I Learning Hub curricula to train novice participants in the multicomponent D4D&I implementation strategy bundle when implementing new programs or innovations. The components of the D4D&I strategy bundle include (i) Pre-implementation assessment for proactive planning and multilevel contextual assessment, (ii) Multilevel partner engagement to learn what is important to end-users and obtain buy-in, (iii) Implementation and adaptations guided by a pre-implementation assessment while retaining program fidelity, and (iv) Program evaluation. We utilized a virtual e-learning platform, expert trainers, mentorship, and a Virtual Learning Collaborative to deliver the six-module curricula to support participants' growth and success. We used quantitative and qualitative methods informed by the Kirkpatrick Evaluation Model to evaluate the D4D&I Learning Hub. Thirty-one participants completed the D4D&I Learning Hub across four cohorts. Participants found the D4D&I curricula relevant and favorable, indicating they acquired the intended knowledge and skills. In presentations of their key takeaways, participants cited a greater understanding of how to apply various D&I theories, models, and frameworks to their research, engage multilevel partners during all phases of implementation and evaluation, and assess fidelity and adaptations. Participants planned to incorporate the acquired D&I knowledge and skills in future publications, grant applications, and when implementing new programs and projects in clinical settings. The D4D&I Learning Hub provides foundational education for novice participants of D&I science. It promotes designing, disseminating, implementing, and evaluating effective programs in clinical settings. This article discusses the development and implementation of the D4D&I Learning Hub. In addition, we evaluated the first four cohorts using the Kirkpatrick Evaluation Model.

Evaluation Of The Flip Learning Program In Improving Students' English Language Ability At The Cibinong Lia Language Institute

Study This analyze effectiveness of flipped learning program in increase ability speak English students at the LIA Cibinong Language Institute. Using approach qualitative descriptive , research This adopting the Kirkpatrick evaluation model which includes four aspect : reaction students , learning process , change behavior , and results end program implementation . Data collected through observation , interviews , and documentation , as well as analyzed through stages condensation , presentation , and withdrawal conclusion . Subject study involving head branch , staff academics , instructors , and student from LIA Cibinong and Jalan Baru, with implementation during May to July 2024. Research results show that implementation flipped learning get response positive from students , improve participation active in discussion class and develop skills think critical . Students also reported improvement ability speak English , especially in speaking , listening , and mastery vocabulary . Although challenge like motivation Study independence and limitations technology Still faced , this program proven effective in increase quality learning and skills social students . Evaluation through observation class confirm existence improvement consistent in skills Language students . In addition , this program impact positive on growth business institutions . In conclusion , flipped learning proven as effective and feasible method​ developed more continue at the LIA Cibinong Language Institute. Recommendations development covers optimization motivation Study through gamification , enhancement quality material , optimization facilities , as well as training addition for teacher For maximize potential method This .

The Influence And Effectiveness of Training on Workforce Performance Using The Kirkpatrick Evaluation Model of PT Pertamina Hulu Rokan

For workers, training, skills, information technology, knowledge, professional ethics are very important for the world of work. The KirkPatrick Evaluation Model attempts to help analyze whether existing training in a company is running effectively. So from this paper, it can be concluded that the training carried out by the company has received a very positive response from the workers so that the results of the evaluation that has been carried out are useful for the company and subsequent training.

Construction of the Index System for the Integration of Professional Education and Innovation Entrepreneurship Education in Applied Universities

In applied universities, innovation and entrepreneurship education (IEE) has entered a new era of comprehensive integration of professional education. However, as a novel educational activity, the evaluation of professional innovation integration still lacks reasonable appraisal standards. This article is based on the basic theory of the Kirkpatrick evaluation model, combined with the innovation and entrepreneurship practice of China's applied universities, this study initially constructs an index system for professional innovation integration in applied universities, estimates and ascertains the weights of different indicators using the Delphi and analytic hierarchy process approach, and thereafter, builds an index system for professional innovation integration in applied universities, which comprises of 4 primary indicators, namely: reaction layer, learning layer, behavior layer, and achievement layer, supported by 9 secondary indicators and 23 tertiary indicators.

Implementation of Entrepreneurship Course in Maritime Higher Education

Entrepreneurship course is one of the subjects given to seventh semester cadets majoring in Sea Transportation and Port Management at the College of Shipping. This study aims to determine the implementation as well as an evaluation for further entrepreneurship learning. The evaluation was conducted using the Kirkpatrick evaluation model developed by Mullins which consists of four aspects: satisfaction, learning, behaviour and impact. The method used is a combination of quantitative and qualitative with data analysis in the form of percentages. The sample was taken from 140 cadets who had completed the entrepreneurship course. Data was collected through a questionnaire administered via google form. The results showed that the level 1 evaluation of cadets' satisfaction in learning showed an average score of 71% satisfied learning entrepreneurship and 29% dissatisfied. From the level 2 evaluation, it was obtained that the average cognitive, affective and psychomotor learning outcomes had increased. Level 3 evaluation of entrepreneurial behaviour obtained an average increase of 20 points. Level 4 evaluation of the impact of learning obtained 82% of cadets think it is suitable to be continued, and 75% think entrepreneurship learning is effective in making cadets become entrepreneurs, and 72% form cadets to carry out business activities.

Supporting Clinical Informatics Competency Development Among the Clinical Informatics Team at Providence Health Care.

Clinical informatics (CI) competencies are crucial for health care organizations to effectively use information communication technologies (ICTs) and deliver quality care. An interdisciplinary CI team can assist organizations with leveraging ICTs, but may also require support. This case study describes a peer-led knowledge translation project designed, delivered and implemented over two years by members of the CI team at Providence Health Care (PHC). The project included CI competencies assessment of CI team members, followed by tailored education for identified knowledge gaps. The Kirkpatrick evaluation model was used to assess three levels of learning among CI team members, including a satisfaction survey, pre-and post-cognitive retention of the education intervention using a validated tool for informatics specialists, and project partner feedback of CI team performance 12 weeks after education completion. This case study provides evidence-informed guidance on 'how to' implement peer-led, practice-based CI training for CI teams.

Operationalizing a Weighted Performance Scoring Model for Sustainable e-Learning in Medical Education: Insights from Expert Judgement

Validation is needed for any newly developed model or framework because it requires several real-life applications. The investment made into e-learning in medical education is daunting, as is the expectation for a positive return on investment. The medical education domain requires data-wise implementation of e-learning as the debate continues about the fitness of e-learning in medical education. The domain seldom employs frameworks or models to evaluate students' performance in e-learning contexts. However, when utilized, the Kirkpatrick evaluation model is a common choice. This model has faced significant criticism for its failure to incorporate constructs that assess technology and its influence on learning. This paper aims to assess the efficiency of a model developed to determine the effectiveness of e-learning in medical education, specifically targeting student performance. The model was validated through Delphi-based Expert Judgement Techniques (EJT), and Cronbach's alpha was used to determine the reliability of the proposed model. Simple Correspondence Analysis (SCA) was used to measure if stability is reached among experts. Fourteen experts, professors, senior lecturers, and researchers with an average of 12 years of experience in designing and evaluating students' performance in e-learning in medical education participated in the evaluation of the model based on two rounds of questionnaires developed to operationalize the constructs of the model. During the first round, the model had 64 % agreement from all experts; however, 100% agreement was achieved after the second round, with all statements achieving an average of 52% strong agreement and 48% agreement from all 14 experts; the evaluation dimension had the most substantial agreements, next to the design dimension. The results suggest that the model is valid and may be applied as Key Performance Metrics when designing and evaluating e-learning courses in medical education.

The Effectiveness of a Psychoeducation Intervention Program for Enhancing Self-Esteem and Empathy : A Study of Emerging Adult College Student

This research examines the effectiveness of psychoeducation intervention programs among emerging adult college students. We used the quantitative method with single-subject designs—Kirkpatrick evaluation model conducted by targeting two measurement levels: reaction and learning. Quantitative data analysis used the Rosenberg Self-Esteem Scale (RSES) and the Basic Empathy Scale (BES). Qualitative data were measured through baseline, treatment, and follow-up assessments. The findings of this research demonstrated that enhancing self-esteem and empathy through this program was effective. Significant changes in the subjects' knowledge and attitudes were observed after the intervention. This research suggests that emerging adult college students can gain a deeper understanding of their potential and experience a positive mind and emotional states through increased self-esteem and empathy.

Evaluating Virtual Simulation to Augment Undergraduate Nurses' Clinical Practice.

Due to the lack of clinical placements during the pandemic, virtual simulation was used to augment student practice experiences. Using Kirkpatrick's evaluation model, a program evaluation study using a mixed-methods design was implemented to assess student and faculty satisfaction and usefulness of virtual simulation, the effectiveness of meeting learning needs, and the effects of the virtual simulation resource on the development of clinical judgment (n = 70). Virtual simulation was rated as moderately useful with an overall mean of 1.7 (SD = 0.66, range 1 to 3). Only 21% to 49% of the students found online simulation either met or well met the various areas of learning needs. Qualitative data highlighted the benefits of this strategy as well as implementation factors that affected students' experience. Virtual simulation can be used in clinical courses to augment learning when implemented in a way that addresses students' needs. [J Nurs Educ. 2024;63(7):470-477.].

Inquiry for the win: Fostering curiosity through a BINGO exercise in a longitudinally integrated clerkship

Educational Challenge As technological advancements challenge the current roles healthcare workers fill, curiosity and lifelong learning are becoming increasingly important. However, descriptions of specific curricular interventions that successfully develop these attitudes and skills remain limited. Proposed Solution We aimed to promote curiosity and advance practical application of evidenced-based medicine through a longitudinal, gamified learning activity within a longitudinal integrated clerkship (LIC). Thirty-seven students across two inquiry-themed LICs based at a university hospital and a community-based integrated healthcare delivery system used BINGO cards containing various assignments designed to develop the skill of asking and answering clinical questions. Assignments included: engaging experts, using evidence-based medical resources, attending education events, utilizing medical library resources, and Phone-a-Scientist. Students shared their experiences in a group setting and wrote a reflection for each assignment that was qualitatively analyzed to determine the perceived educational outcomes according to the Kirkpatrick Evaluation Model. Lessons Learned Results suggest that Inquiry BINGO fosters curiosity and builds skill in applied evidenced-based medicine early in clinical training. Most assignments prompted students to engage in opportunities they might not have otherwise explored. All but three students reported a change in behavior as a result of the assignment and 57% reported positive clinical or research results. Next Steps Future iterations of this activity should be evaluated by obtaining patient and/or preceptor feedback as well as longitudinal evaluations of student behavior to ensure higher level educational outcomes are being met.

Assessment of an In-Service Training Activity Transformed into an E-Learning Environment Using the Kirkpatrick Model

The aim of this study is to evaluate an in-service training program transformed into an e-learning environment using the Kirkpatrick evaluation model. A single-group experimental design was employed in the research. Thirty teachers who participated in the in-service training program were included in the study. The program was adapted to the e-learning platform and presented to the participants. Data were collected through surveys. 90% of the participants expressed satisfaction with the program, finding the e-learning format beneficial, and liking the content and presentation of the program, stating that it helped reinforce their knowledge. They mentioned a more flexible and comfortable learning experience compared to face-to-face training and acquired the targeted knowledge and skills by the end of the program. The findings indicate that an in-service training program delivered face-to-face can be successfully transformed into an e-learning environment and evaluated using the Kirkpatrick evaluation model. The e-learning format provided participants with a more flexible and comfortable learning experience. The program significantly improved participants’ knowledge, skills, and behaviors. Adapting the program to similar professional groups can offer various benefits, such as enhancing professional skills, increasing workplace productivity, and improving professional satisfaction.

A systematic review of shared decision making training programs for general practitioners

BackgroundShared decision making (SDM) has been presented as the preferred approach for decisions where there is more than one acceptable option and has been identified a priority feature of high-quality patient-centered care. Considering the foundation of trust between general practitioners (GPs) and patients and the variety of diseases in primary care, the primary care context can be viewed as roots of SDM. GPs are requesting training programs to improve their SDM skills leading to a more patient-centered care approach. Because of the high number of training programs available, it is important to overview these training interventions specifically for primary care and to explore how these training programs are evaluated.MethodsThis review was reported in accordance with the PRISMA guideline. Eight different databases were used in December 2022 and updated in September 2023. Risk of bias was assessed using ICROMS. Training effectiveness was analyzed using the Kirkpatrick evaluation model and categorized according to training format (online, live or blended learning).ResultsWe identified 29 different SDM training programs for GPs. SDM training has a moderate impact on patient (SMD 0.53 95% CI 0.15–0.90) and observer reported SDM skills (SMD 0.59 95%CI 0.21–0.97). For blended training programs, we found a high impact for quality of life (SMD 1.20 95% CI -0.38-2.78) and patient reported SDM skills (SMD 2.89 95%CI -0.55-6.32).ConclusionSDM training improves patient and observer reported SDM skills in GPs. Blended learning as learning format for SDM appears to show better effects on learning outcomes than online or live learning formats. This suggests that teaching facilities designing SDM training may want to prioritize blended learning formats. More homogeneity in SDM measurement scales and evaluation approaches and direct comparisons of different types of educational formats are needed to develop the most appropriate and effective SDM training format.Trial registrationPROSPERO: A systematic review of shared-decision making training programs in a primary care setting. PROSPERO 2023 CRD42023393385 Available from: https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42023393385.

Implementasi Evaluasi Model Kirkpatrick Pada Program Pendidikan Karakter Institut Agama Islam Sultan Muhammad Syafiuddin Sambas

The aim of this research is to analyze using the Kirkpatrick evaluation model with 4 levels (Kirkpatrick Four Levels Evaluation Model) which aims to determine the effectiveness of the Character Education program at the Sultan Muhammad Syafiuddin Sambas Islamic Institute. This research method uses a qualitative descriptive qualitative research method. Data collection data was obtained through observation instruments, interviews and documentation. The results of this research are that the assessment at level 1 of reactions, the results of participants' reactions to the organizing committee and resource persons are very high. You just need to pay attention to the preparation of learning materials that need to be prepared at the beginning. Education and stationery equipment such as notebooks need to be checked again. At level 2 of learning, the participants' attitudes, knowledge and skills are very significantly assessed. At behavioral level 3, the evaluation is carried out after the participants return to their respective classes. There is an increase in changes in behavior of Pendikar Participants from before Pendikar and after PENDIKAR. Changes in behavior such as attendance discipline, how to dress, motivating colleagues, communicating well, as well as accuracy and speed in completing daily tasks. At level 4 impact, there is an increase in the learning of PENDIKAR participants. Participants are able to guide and disseminate the knowledge they have gained from PENDIKAR to their fellow colleagues, are able to develop learning methods and media in the learning process on campus, and their behavior becomes better

Implementation of the Kirkpatrick Model in Evaluating the Use of Digital Technology in Counseling Guidance Programs at SMP Negeri 1 Matan Hilir Utara

This research aimed to evaluate the Guidance Counseling (BK) service program in the use of digital technology at SMP Negeri 1 Matan Hilir Utara using the Kirkpatrick Evaluation Model. This research method was quantitative with a descriptive approach. Data is technically collected using questionnaire techniques which are the main data source. The subjects in the research were 60 class VIII students who were determined based on a saturated sampling technique where all members of the population were used as research samples. In the context of this research, students act as evaluators or beneficiaries, while the guidance and counseling teachers and school principals are the parties who evaluate the guidance and counseling program services that are implemented. The results of the research show that: 1) At the reaction level, BK services are perceived as showing a scale using digital technology that is on a medium scale with an average of 3.35, 2) At the learning level, BK services in terms of technology use obtain an average of 3. 57 in the high category, 3) At the behavior level with indicators of application of values in the program it is in the medium category with an average of 3.38 in aspects of technology use that need special attention to be optimized, 4) At the results level, students assume that the use of technology is in the medium category with an average of 3.35 which needs to be maximized to improve the quality of the BK program at the research location.

Study of The Technical Index of Online Learning Behavior Analysis of Nursing Majors on The Superstar Platform Based on The Kirkpatrick Evaluation Model

Objective: To analyze the technical indexes of students’ online learning behavior analysis based on Kirkman’s evaluation model, sort out the basic indexes of online learning behavior, and extract scientific and efficient evaluation indexes of online learning effect through statistical analysis. Methods: The online learning behavior data of Physiology of nursing students from 2021–2023 and the first semester of 22 nursing classes (3 and 4) were collected and analyzed. The preset learning behavior indexes were analyzed by multi-dimensional analysis and a correlation analysis was conducted between the indexes and the final examination scores to screen for the dominant important indexes for online learning effect evaluation. Results: The study found that the demand for online learning of nursing students from 2021–2023 increased and the effect was statistically significant. Compared with the stage assessment results, the online learning effect was statistically significant. Conclusion: The main indicators for evaluating and classifying online learning behaviors were summarized. These two indicators can help teachers predict which part of students need learning intervention, optimize the teaching process, and help students improve their learning behavior and academic performance.

Evaluation of the one-minute preceptor as a teaching method for family medicine residents

Background and objective: Health professionals struggle to appropriately train resident doctors in overcrowded medical centers. The one-minute preceptor teaching approach is an effective teaching model that provides a solution to the problem. This study aims to determine the effectiveness of a one-minute preceptor in improving the clinical teaching skills of supervisors and the critical thinking skills of residents and to assess the residents' and supervisors’ attitudes toward a one-minute preceptor. Methods: From October 1, 2021, to February 28, 2022, a cross-sectional study was conducted in the Family Medicine department at Hawler Medical University, in Erbil, Iraq. Eight family medicine supervisors and 30 resident doctors at the specialized Family Medicine Health Center were involved in the study. Pre-and-post assessment questions were administered to the supervisors before and after one-minute preceptor training to assess their knowledge. Kirkpatrick's evaluation model was used to evaluate supervisors’ knowledge and application of skills after training, and to assess whether intended outcomes occurred with the residents. Results: All supervisors agreed that this approach to teaching should be incorporated into medical education (50% agreed and 50% strongly agreed) and improved their teaching skills (75% agreed, 25% strongly agreed). More than half of the residents agreed that this new teaching method should be incorporated into medical education. (50% agreed and 10% strongly agreed) Conclusion: The one-minute preceptor teaching model intends to provide doctors with an effective framework to improve their instructional effectiveness.