The effectiveness of dialectical behaviour therapy training: a quantitative systematic review using Kirkpatrick's four-level model.

Integrating assistive technology (AT) into occupational therapy (OT) education is critical to educating students to utilize AT to enhance individual independence. This study aims to describe the development of a science, technology, engineering, arts, and mathematics (STEAM)-based AT curriculum in OT and evaluate its effectiveness using the Kirkpatrick model. An 18-week STEAM-based AT curriculum was created with lectures, hands-on activities, and a simulation exam. A one-group pretest-posttest observational study was undertaken with 40 undergraduate students to gauge its effectiveness. Their professionalism was evaluated using a multidimensional checklist, whereas a satisfaction survey gathered their feedback on the curriculum experience. Level 1: Participants reported high satisfaction (mean = 4.61, SD = 0.36), with a 97.5% agreement. Level 2: 95% (n = 38) passed the AT curriculum exam. Level 3: Therapists and patients expressed satisfaction with participants' AT skills (therapists: t(39) = 12.22, p < 0.001; patients: t(39) = 8.84, p < 0.001). Level 4: Participants showed significant improvement in professionalism (t(39) = 16.45, p < 0.001). Lastly, using the Kirkpatrick model, the STEAM-based AT curriculum passed the evaluation. The STEAM-based AT curriculum in OT enhances professionalism and potentially boosts patient satisfaction with AT services provision.

This study aimed to evaluate an internship programme for preservice teachers. The program evaluation model uses the Kirkpatrick Model which consists of four levels: reaction, learning, behaviour, and outcome. This model is suitable for evaluating internship programmes in reviewing the development of preservice teachers from the initial level to the completion of the internship programme targets. This study involved prospective teachers (n=205) in Economics and Business at the Universitas Negeri Medan. Data were collected online using a Google Forms questionnaire over two months in 2022. Data analysis used quantitative description to explore the programme achievements at each level in depth. The findings showed that the mean of reaction (5.05), learning (5.20), behaviour (4.96), and outcome (5.03) were above the ideal mean of 3.50. Internship programme participants have met the average achievement target of Kirkpatrick's four-level model, where learning evaluation is more meaningful for student teachers when participating in the internship programme compared to other levels. Overall, with this research, it is hoped that internship programme managers can design more targeted and measurable programmes to improve the experience of prospective teachers. Keywords: internship program, preservice teacher, evaluation, Kirkpatrick four-level

The development of artificial intelligence (AI) provides new opportunities for improving teachers' professional competencies, particularly in the development of digital learning tools. However, the implementation of AI-based training in coastal schools still faces obstacles, mainly related to infrastructure limitations and digital literacy. This study aims to evaluate the effectiveness of the Teacher Capacity Building Programme in the Use of AI for the development of learning tools at SMA Negeri 2 Bintan Pesisir, Riau Islands Province. The evaluation was conducted using Kirkpatrick's four-level model, which includes reaction, learning, behaviour, and results. This study used a mixed methods approach with data collection through questionnaires, pretest–posttest tests, and observation. The results showed that the training programme obtained an overall average score of 84.3, which is in the good to very good category. Specifically, the participants' reaction level scored 86.8, learning level 84.4, behavioural change 80.3, and final results 85.7. Correlation analysis between levels showed a coefficient of 0.87, indicating a strong linear relationship between evaluation components. These findings indicate that AI-based training contributes positively to improving teachers' ability to design digital learning tools, although its implementation is still influenced by contextual factors in schools. Theoretically, this study reinforces the relevance of the Kirkpatrick model in evaluating digital training. Practically, the results of this study can be used as a reference in designing technology-based teacher training programmes, especially in coastal areas and regions with digital limitations.

Mo, J.; Mao, C.M., and Zuo, Z.Y., 2020. Training effectiveness evaluation system of migrant workers in coastal cities. In: Guido Aldana, P.A. and Kantamaneni, K. (eds.), Advances in Water Resources, Coastal Management, and Marine Science Technology. Journal of Coastal Research, Special Issue No. 104, pp. 984–987. Coconut Creek (Florida), ISSN 0749-0208.Based on Kirkpatrick's four-level model ,combined with the principle of index selection and the characteristics of migrant workers coastal cities, the paper initially establishes the evaluation index of training effect on the four levels of reaction layer, learning layer, behavior layer and result layer. Using the method of AHP and fuzzy comprehensive to evaluate the training effect of migrant workers along Yangtze River valley. Finally, the paper put forward the way to improve the training effectiveness of migrant workers along Yangtze River valley.

As e-training has shown great promise, Microsoft (China) and the Ministry of Education of China (MOE) have launched e-training for in-service teachers in order to improve teachers' instructional skills. 16,264 primary and secondary school teachers participated in this e-training. This study aims to evaluate the effectiveness of the e-training using the Kirkpatrick's four-level model. Discourse data, testing scores, homework scores, and final passing rate were collected and analyzed. The results indicated that the e-training indeed improved trainees' instructional skills. The final passing rate of e-training was over 80%. The results also indicated that it was appropriate to use the Kirkpatrick's four-level model to evaluate the e-training.

Medical students face high levels of psychological distress but often hesitate to seek formal support. Peer support programmes offer a student-centred approach to fostering well-being and resilience. This study examines the educational impact of the Ramathibodi Peer Support Programme (Rama PSP), a student-led initiative in Thailand, with a focus on the development of peer supporters. Thirty-seven medical students across all six years participated in a 10-week training programme. A mixed-methods design was employed, guided by Kirkpatrick’s Four-Level Model of Training Evaluation. Quantitative data were collected using the mid- and end-course assessments, Rama PSP aptitude assessment, PHQ-9 and GAD-7. Qualitative data were obtained through post-training interviews and analysed using content analysis. Peer supporters demonstrated self-reported significant improvements in key competencies, including clarifying questions, referrals and boundary-setting. No significant changes were observed in PHQ-9 and GAD-7 levels, consistent with the programme’s educational–not therapeutic–focus. Qualitative findings revealed substantial growth in self-awareness (96.2%), self-esteem, self-management and social awareness. Peer supporters reported applying their skills in both formal and informal peer interactions. Rama PSP suggests that structured peer support training may be associated with perceived improvements in interpersonal competencies and developmental outcomes among medical students. By focusing on peer supporters, the programme may support professional identity formation and complement traditional curricula. Integrating such training could help foster empathy, resilience and a supportive learning environment; however, further research using objective measures is needed.

Education and training interventions can be evaluated through the success of learning outcomes. Kirkpatrick's four-level model is a widely accepted and highly popular evaluation tool. However, some criticise the model's shortcomings. This article will examine the extent to which the four-level model can evaluate design and technology students' learning about aesthetics after an intervention by reporting our use of an augmented version of the four-level model. We examine the results in terms of students' reaction to the intervention, their long-term learning and their behaviour changes by studying their visual analyses and drawings through segment codes. We found that, in order to uncover the obscurities imbedded in aesthetics and to explicate the complexities, we could not use the four-level-model on its own, but had to revert to a more augmented version.

Donald Kirkpatrick developed the basis of his four-level model in 1954, which he published in four articles for the first time in 1959. It is today one of the most acknowledged models for evaluating training programmes. The levels are: reaction (participant satisfaction), learning (learning success), behaviour (learning transfer) and results (business success). Kirkpatrick assume causal linkages between the four levels. Since the 1960s, these assumptions were tested with little evidence. In the presented study, 43 training courses with altogether 335 participants were evaluated on the levels (1) reaction, (2) learning and (3) behaviour. The data show no correlation between reaction and learning and also no correlation between reaction and behaviour. In view of this result, the practice of evaluating professional trainings based on participant satisfaction requires further development.

A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence

e-Learning is one way to provide continuing education and has been shown to be an effective method. However, little is known about how psychiatric organizations benefit from continuing education. The aim of this study was to use Kirkpatrick's four-level model to evaluate an e-learning continuing education course from the perspectives of nursing managers in psychiatric hospital organizations. This qualitative design evaluated one e-learning course using Kirkpatrick's model as an evaluation frame. Nursing managers noticed positive, neutral, and negative reactions after the course. Nursing managers also reported that the nurses' learning evolved.

Are virtual reality intravenous injection training programs effective for nurses and nursing students? A systematic review

This Guide reviews theories of science that have influenced the development of common educational evaluation models. Educators can be more confident when choosing an appropriate evaluation model if they first consider the model's theoretical basis against their program's complexity and their own evaluation needs. Reductionism, system theory, and (most recently) complexity theory have inspired the development of models commonly applied in evaluation studies today. This Guide describes experimental and quasi-experimental models, Kirkpatrick's four-level model, the Logic Model, and the CIPP (Context/Input/Process/Product) model in the context of the theories that influenced their development and that limit or support their ability to do what educators need. The goal of this Guide is for educators to become more competent and confident in being able to design educational program evaluations that support intentional program improvement while adequately documenting or describing the changes and outcomes—intended and unintended—associated with their programs.

To evaluate the educational outcomes of a nationwide faculty development initiative tailored to obstetrician-gynecologists in Japan, addressing the gap in systematic faculty development implementation and assessment within the specialty. This observational study evaluated three online faculty development seminars conducted by the Japan Society of Obstetrics and Gynecology between December 2023 and August 2024. Using Kirkpatrick's four-level model, we assessed participant satisfaction (Level 1), self-assessed knowledge gain (Level 2), and behavioral implementation in clinical practice (Level 3). Data were collected using post-seminar and longitudinal follow-up questionnaires. Level 1 (Reaction): Among the 195 participants, 85.1% completed the post-seminar surveys, with satisfaction rates ranging from 87.9% to 93.0%. Level 2 (Learning): Self-assessed knowledge scores significantly improved across all three seminars (e.g., Seminar 1: from 4.0 to 7.0 on a 9-point scale, p < 0.05). Level 3 (Behavior): Among participants who attended multiple sessions, 74.1% reported implementing learned skills, such as feedback techniques and leadership strategies, with the majority noting positive educational outcomes. Nationwide faculty development programs tailored to obstetrics and gynecology can enhance educational competence and promote behavioral change. Our findings highlight the feasibility and effectiveness of specialty-specific faculty development and indicate the need for long-term objective assessments to ensure a sustainable impact. These findings have significant implications for the field of obstetrics and gynecology, suggesting that targeted faculty development initiatives can play a crucial role in improving the quality of medical education and addressing workforce shortages.

Health sciences education (HSE) must innovate to meet the demands of an increasingly complex healthcare system. Despite increasing sophistication, there is a critical gap in intentional training in innovation, change management, and leadership. To address this, the Research, Innovation, and Scholarship in Education Innovator Development Program was launched at Michigan Medicine in 2019 to cultivate innovation as a professional competency. Program design was informed by Kotter's Change Management framework and a framework for health science education innovation. A literature review and modified Delphi process with 24 faculty, staff and learners, prioritized seven core competencies, embedded in a 12-month curriculum. Innovators engage in monthly workshops, coaching, and consultations with team members, applying new concepts to interventions across three phases: pre-implementation, pilot, and iteration/scaling. This linking of theory, practice, and individualized feedback represents a novel approach to developing innovation capacity and leadership. From 2019 to 2025, four cohorts (N = 31) completed the program, with 10 currently enrolled, representing faculty, learners, and staff from 18 departments. Outcomes were evaluated using Kirkpatrick's four-level model. At Level 1 (reaction), participants rated all components as valuable, with coaching and workshops highest. At Level 2 (learning), self-reported knowledge gains were significant across all competencies (p < .001, large effect sizes). At Level 3 (behavior), self-assessed scores increased significantly, but team ratings were mixed, with teamwork and curiosity declining. At Level 4 (results), 27 interventions were piloted and 13 scaled beyond the pilot. Institutional support, cognitive diversity, and intentional competency development contributed to success. The IDP demonstrates institutional feasibility, though broader feasibility requires further study given resource needs. Next steps include strengthening competencies with inconsistent growth, refining assessments, and longer-term follow-up to evaluate scholarly productivity, career advancement, and dissemination. The program offers a structured approach to building innovation capacity, though higher-level impacts remain limited.

Mealtime assistance is essential in long-term care for older adults with dementia, as it promotes autonomy, social interaction, and dignity. Flexible and accessible training for direct care workers (DCWs) is crucial in improving mealtime support. This study aimed to develop and evaluate the effectiveness of a YouTube-based mealtime assistance education program for dementia care based on the cognitive theory of multimedia learning. The intervention was designed to provide accessible and flexible training for DCWs in care facilities. A quasi-experimental, mixed-methods design was used. Twenty-four DCWs and five supervising nurses from two long-term care facilities participated in this study. To maximize participation, DCWs were non-randomly assigned to the intervention based on facility recommendations and individual work schedules. The effectiveness of the program was evaluated using Kirkpatrick's four-level model. Reactions were measured by collecting satisfaction data from the DCWs through surveys and focus group interviews (FGIs). Learning outcomes were assessed using self-reported questionnaires to measure mealtime assistance knowledge with pre- and post-intervention comparisons. Behavioral changes were evaluated by supervising nurses using an observation-based questionnaire and analyzed using pre- and post-intervention data. Organizational results and the dissemination effect were explored through FGIs with nurses and thematically analyzed. The median mealtime assistance knowledge score increased significantly (Pre: 91.67, Post: 100.00, p = .002, effect size = 0.45, interquartile range: 83.3-97.92). The median mealtime assistance behavior score showed significant improvement (Pre: 67.00, Post: 92.00, p < .001, effect size = 0.56, interquartile range: 82.33-92.00). Supervising nurses observed enhanced ethical attitudes and behavioral changes among DCWs during mealtime assistance. YouTube-based multimedia learning interventions effectively improved the knowledge, attitudes, and behaviors of DCWs in mealtime assistance for older adults with dementia. Its accessibility and practicality suggest that this approach can be a useful educational model in long-term care settings. Further studies are needed to explore the role of opinion leaders in sustaining and expanding these effects.