Kirkpatrick's Four-level Evaluation Model Research Articles

The Effectiveness of Technology-Enhanced Learning in Prosthodontic Education: A Systematic Review.

To compare the effectiveness of technology-enhanced learning (TEL) with traditional learning in prosthodontic education. The protocol was registered before commencement [CRD42023390277]. The search was conducted to identify randomized controlled trials (RCTs) focused on comparative assessment of TEL and traditional learning in prosthodontic education up to 6 June 2024. A comprehensive search strategy was formulated, encompassing multiple electronic databases. Screening, data extraction and risk of bias assessment using the Cochrane tool (RoB 2) were performed. Educational outcomes were evaluated by Kirkpatrick's four-level evaluation model. Of 2830 records identified, 14 RCTs were included. All the trials were judged at high risk of bias or had some concerns. The included studies assessed the short-term benefits of various TEL approaches, including e-learning, blended learning, video demonstration specialized self-evaluation software and virtual reality simulation in prosthodontic education for undergraduates. Promising results were found in individual studies, particularly in terms of perceived learning conditions with problem-based learning (PBL) and self-reported understanding and confidence when using video demonstrations. Most studies reported comparable outcomes to traditional learning in practical skills assessments, while the results regarding the acquisition of theoretical knowledge through the incorporation of TEL approaches were contradictory. One study reported promising results in perceived learning conditions with the PBL approach, as well as improved self-reported understanding and confidence with the use of video demonstrations. Furthermore, there was a limited number of studies assessing the effect of TEL approaches on students' behaviour. There is a lack of high-quality evidence to support the adoption of TEL approaches in prosthodontics. However, the incorporation of virtual reality simulation has demonstrated promising results, particularly in improving practical clinical skills.

Evaluation Method of Mixed Teaching Efficiency of College Teachers Based on Kirkpatrick Model.

Online teaching has become more prevalent in recent years as a result of the process of digitization and adaptation to the times. Classroom instruction is no longer just conducted in person offline. Online education has received a lot of promotion, especially in the wake of the epidemic's effects, and has evolved into a blended teaching model that combines the two modes. The old system of measuring the effectiveness of teaching is no longer reliable or scientific in assessing the new blended teaching model, and it has many flaws. A perfect system for evaluating the effects of blended learning must therefore be built. In order to more accurately assess the teaching impact of the blended teaching model, this study aims to build a more comprehensive system for evaluating blended teaching effects. This paper introduces the Kirkpatrick four-level evaluation model, which plays a significant role in the evaluation field, in order to achieve this goal. The experiments in this study compared the Kirkpatrick model with the AHP, which is frequently used in the evaluation. Online, offline, and blended teaching models were the subject of an experimental investigation. The experimental results demonstrated that each index combination's weights in the Kirkpatrick model and the AHP model of the blended teaching effect evaluation system were fairly close to one another. The evaluation indicators all center on the assessment of practical ability after class, which is consistent with their focus. The innovation evaluation of course papers has the highest weight, and the weight of evaluation indicators is greater than 0.15; the Kirkpatrick model has the lowest weight of the number of platform logins, with a weight value of 0.011, and the evaluation index of the teaching effect evaluation system based on AHP.

Kirkpatrick Four-level Model Evaluation: An Evaluation Scale on the Preservice Teacher’s Internship Program

Internship programs are organized depending on the program provider (school), so many students have a big difference in the internship experience. On the other hand, the campus has limitations in running the program, one of which is evaluating the achievement of the internship program. The purpose of this study is to build validity on the evaluation scale of the apprenticeship program for pre-service teachers using Kirkpatrick's four-level evaluation model. This type of research is quantitative with a confirmatory quantitative approach. The sample is 212 pre-service teachers. The methods used in collecting data are observation, tests, and questionnaires. Instruments used for test sheets and questionnaires. Data analysis used a confirmatory factor analysis (CFA) approach with the SmartPLS v3.0 application. The analysis results are that the four-level evaluation model meets construct validity and proves accurate validation on the internship program evaluation scale. In particular, the internship program evaluation scale effectively tracks program achievements at each level. The implication is that academic stakeholders on campus can use this evaluation scale as a measuring tool to experiment with the success of the internship program. In addition, they can review the achievements of pre-service teachers in implementing apprenticeship programs based on each level of the Kirkpatrick model.

"This Project Helped Me to Grow": Experiences of Teaching Palliative Care Using the EDUPALL Project Medical Undergraduate Curriculum.

Background: The WHO has proclaimed that palliative care (PC) should be integrated as a routine element of all undergraduate medical and nursing education. The EDUPALL Erasmus+project produced a PC curriculum for undergraduate medical education based on the European Association for Palliative Care (EAPC) recommendations for undergraduate training. This was tested in four Romanian Faculties of Medicine: Universities of Transilvania, Iasi, Targu Mures, and Timisoara. The aim of this study is to describe teachers' satisfaction and views on the effectiveness of the EDUPALL curriculum and supporting learning materials. Methods: We conducted nine semistructured interviews with teachers involved in EDUPALL implementation in their universities. Interviews were transcribed and collected data underwent thematic analysis. Kirkpatrick's four-level evaluation model of training was employed to synthesize the outcomes into final categories of reaction, learning, behavior, and results. Results: Data were categorized against Kirkpatrick's four levels as follows: Level 1 (Reaction) EDUPALL curriculum-a good standard with achievable goals; Level 2 (Learning) Personal appraisal and development needs of the teaching faculty; Level 3 (Behavior) Application of competencies and student feedback; and Level 4 (Results): Faculty- and country-level Impact of the EDUPALL project. Conclusion: EDUPALL curriculum is a good and adaptive model to teach PC at Faculties of Medicine, considered by teachers as a way of bridging an existing training gap for medical students in building essential competencies in symptom management, communication, spirituality, and self-awareness.

Applying the Kirkpatrick model to evaluate dental students' experience of learning about antibiotics use and resistance.

"Antibiotic resistance" is of main concern in global health and that it could hinder the achievement of sustainable development goals. One of the reported contributing factors is the irrational prescribing behaviour of healthcare professionals including dentists. Efforts to design and evaluate effective educational programmes for undergraduate dental students about appropriate prescribing behaviour during their early educational years could mitigate the risk of antibiotic resistance. A total of 322 students participated in the study. Their knowledge of antibiotics and antibiotic resistance was assessed. Students received a specially designed, interactive educational programme. The success of the programme was assessed after the educational sessions and 2years later using Kirkpatrick's four-level evaluation model. This study is the first study to use Kirkpatrick model to evaluate an educational programme for dental students. The results of the first level of evaluation showed an overall satisfaction score of 82.1%. The second level revealed an increase in the percentage of correct answers after the educational sessions from 68.3% to 80%, and significant agreement with responsible antibiotics usage (p-value=.020, Effect size=0.121). The third level showed that the percentage of correct answers 2years later was 87.5%. The fourth level confirmed the success of the programme as 88.9% of participants reported using the knowledge gained from the programme when prescribing antibiotics. Our study emphasises the success of the used educational programme and highlights the need for educational interventions in the under graduate dental curriculum.

On the Record: An Innovative Simulation Teaching Resource to Teach Undergraduate Nurses How to Write Legally Defensible Patient Documents.

Good patient documentation writing skills are an essential component of a nurse's portfolio to satisfy the professional and legal requirements of record keeping. Traditionally, however, documentation is viewed as a dry topic as it is not always applied well to practical situations; therefore, this topic readily lends itself to being delivered in an innovative and engaging way. This pilot study used a low fidelity simulated "courtroom" session to teach this essential nursing skill to undergraduate nurses. The pilot study took place with three cohorts of undergraduate nurses (n = 200) from all four fields of nursing in the second year of a 3-year Bachelor of Science nursing program. These results were captured in a variety of ways and evaluated using Kirkpatrick and Kirkpatrick's four-level evaluation model. The results indicated that the simulation engaged and focused students in an interactive way. The pilot study results indicated that a move away from traditional didactic teaching to a more customized method of learning accommodates the documented trends of learning styles across different generations. This pilot study has led to a university-wide study involving students and staff from three different faculties. [J Nurs Educ. 2021;60(7):414-418.].

Evaluation of the B.strong Queensland Indigenous Health Worker Brief Intervention Training Program for Multiple Health Risk Behaviours.

Queensland’s B.strong brief intervention training program was a complex intervention developed for Aboriginal and Torres Strait Islander health workers to assist clients address multiple health risks of smoking, poor nutrition and physical inactivity. This study evaluates program effectiveness by applying the Kirkpatrick four-level evaluation model: (1) Reaction, participants’ satisfaction; (2) Learning, changes in participants’ knowledge, confidence, attitudes, skills and usual practice; (3) Behaviour, application of learning to practice; and (4) Results, outcomes resulting from training. A retrospective analysis was conducted on data for respondents completing pre-training, post-workshop and follow-up surveys. Changes in domains such as training participant knowledge, confidence, attitudes, and practices between survey times were assessed using paired-samples t-tests. From 2017–2019, B.strong trained 1150 health professionals, reaching targets for workshop and online training. Findings showed statistically significant improvements from baseline to follow-up in: participants’ knowledge, confidence, and some attitudes to conducting brief interventions in each domain of smoking cessation, nutrition and physical activity; and in the frequency of participants providing client brief interventions in each of the three domains. There was a statistically significant improvement in frequency of participants providing brief interventions for multiple health behaviours at the same time from pre-workshop to follow-up. Indigenous Queenslander telephone counselling referrals for smoking cessation increased during the program period. B.strong improved practitioners’ capacity to deliver brief interventions addressing multiple health risks with Indigenous clients.

Using Brain Science Theory to Analyze the Unity between Language Input and Output: Methodology Improvement Substantiation

Introduction. Based on the brain science theory of “how people learn” and in order to modernize the methodology of psycholinguistic research, this research used documentary analysis and addressed the standpoint that the 4MAT Teaching and Learning Model can be subsumed into or superimposed on the Kirkpatrick Four-Level Evaluation Model, and vice versa. Meanwhile, the phase of language input and output is analyzed on the basis of the two Models above. In the end, some implications arise so as to provide reference for prospective researchers and practitioners in psycholinguistics.
 The aim of the study. The 4MAT Teaching and Learning Model and the Kirkpatrick Four-Level Evaluation Model are both widely applied, so a deliberate literature review to clarify the integration and the unity between them is conducted that expects to make some theoretical references inspired by the unity available to a wide range of linguistic teaching design and learning performance evaluation.
 The authors argue that the references interconnect teaching design and learning performance evaluation in light of language input and output and therefore help linguistic teachers/trainers with a whole and valid scheme at the very beginning of student learning, and this is the unity that also corresponds to Kirkpatrick & Kirkpatrick’s standpoint: “The end is the beginning”.
 Research methods. The study was conducted using the semantic differential scaling and the method of documentary analysis.
 Results. A combination of brain science theory and Fractal Information Theory has verified initially how the 4MAT Teaching and Learning Model and the Kirkpatrick Four-Level Evaluation Model subsume and superimpose in terms of the theoretical framework, i.e., the unity between a teaching theory and a learning performance evaluation theory. Such integration not only originates from the inherent unity verified by a thoughtful literature review but also receives theoretical support from interdisciplinary studies. Meanwhile, this integration is intertwined with language input and output in a psycholinguistic/neurolinguistic manner.
 Conclusions. A primary investigation using brain science theory and other theories to analyze the integration between the 4MAT Teaching and Learning Model and the Kirkpatrick Four-Level Evaluation Model shows the unity between both models. This investigation led to achieving the purpose of the study: modernizing the methodology of psycholinguistic research. With implementing the components/stages of language input and output as this article proposed, it is expected to be promising in extending and applying both models theoretically and practically in linguistics and other relevant areas in the future. As it comes to studies, it is recommended that the two Models be connected to analyze more teaching models and/or learning performance evaluation models for unity, inquire performance evaluation in collaborations with scenarios in practice, or even associate other disciplines under the implementation of Fractal Information Theory. A possible suggestion for psycholinguistic researchers is to design curricular and lessons based on the Unified Models (Figure 1 & 2) proposed in this study and evaluate instructional efficacy and student learning performance. Another potential research direction is to use each quadrant of the Unified Models and analyze related components in more specific language input and output phases: listening, reading, speaking, writing, and even smaller components in the four types of language skills. As it comes to practice, especially in psycholinguistics and/or other relevant disciplines, the key to apply the two target Models simultaneously depends on how to regulate respective quadrants/levels pro rata as well as the wholeness between them to simultaneously achieve “dynamic equilibrium” in the 4MAT Teaching and Learning Model and “The end is the beginning” in the Kirkpatrick Four-Level Evaluation Model.

Fostering effective use of ICT in agricultural extension: participant responses to an inaugural technology stewardship training program in Trinidad

ABSTRACT Purpose: This study sought to investigate agricultural extension practitioners’ response to an in-service training program offered by the University of the West Indies intended to foster effective use of ICTs (information and communication technologies) through a technology stewardship approach. Design/Methodology/Approach: This study reports on participants responses to an in-service training course on technology stewardship. Data were collected using a pre-course survey, formal course evaluation, classroom observation, and semi-structured interviews with participants. Kirkpatrick's four-level evaluation model was used to structure analysis of the results. Findings: Findings from this study show high levels of receptivity to the technology stewardship training course, that learning objectives of the course are achievable when offered as an in-service training program, and that most participants indicated an intent to apply their learning in a post-course activity. Results from the study raise a number of considerations for future course design and institutional policy in order to better support technology stewardship in practice. Practical implications: Technology stewardship training shows promise as a leadership-oriented approach for expanding the effective use of ICTs among communities of practice for agricultural extension in the Caribbean and elsewhere. Theoretical implications: This study applies key principles of social learning theory from a communities of practice approach with theoretical implications for the design of in-service training related to ICT adoption and use for agricultural extension education and policy. Originality/value: This study reports on the first known application of the technology stewardship model from the communities of practice literature to agricultural extension in the Caribbean.

Impact of CSO Manage Training Programs on Individual and Organizational Performance

Human resource problems faced by CSOs are the limited number of employees compared to organizational activities, low employee skills, and lack of employee experience in managing the activities of the organization. Based on these problems, NICE Indonesia conducted a CSO Manage training program which was participated by 10 CSOs. This research was conducted with the aim of evaluating the CSO Manage training program. 10 CSO participants in the CSO Manage training program were taken entirely as research samples. Evaluation of the CSO Manage training program is carried out using the Kirkpatrick four-level evaluation model. The results showed that the objectives of the CSO Manage training program could be achieved as indicated by an increase in knowledge and changes in participant behavior, as well as an increase in individual and organizational performance. Keywords: Training, CSO Manage, NICE Indonesia, Kirkpatrick Four-Level Evaluation Model DOI : 10.7176/EJBM/12-2-05 Publication date: January 31 st 2020

Strengthening teamwork capability in allied health: implementation of a team development program in a metropolitan health service.

Objective Collaborative practice is critical to optimising patient outcomes in contemporary healthcare settings. Evidence suggests interprofessional learning is an effective way to develop teamwork capabilities, yet these skills are traditionally developed in professional silos, or not at all. This study evaluated the implementation of a team development program, the Team Management Systems (TMS) program, for allied health staff within a large metropolitan health service. Methods A mixed-methods audit-quality improvement study was conducted, using Kirkpatrick's four-level evaluation model to structure evaluation of the program. Semistructured questionnaire and workforce survey data were retrieved immediately, 6 months and 1-2 years after training and applied to each level of the model (Reaction, Learning, Behaviour, Results). Results In all, 886 staff participated in the TMS program from 2014 to 2018. High satisfaction with the program was observed. Knowledge of what constitutes effective teamwork improved significantly (P=0.008) in TMS participants compared with a matched untrained cohort. Participants reported positive behaviour change and continued engagement with TMS principles 6 months after training. Perceived impact of the program on patient and/or organisational outcomes was evident, although less compelling than the changes to knowledge and behaviour. Conclusions The TMS program yielded positive effects on staff satisfaction, knowledge, team dynamics and team behaviours. These findings demonstrate the significant value of such initiatives to enhance the capability and effectiveness of interdisciplinary healthcare teams. What is known about the topic? Complex conditions, increasing comorbidities, specialisation and scarcity of resources mean healthcare workers need to work effectively in teams to achieve quality, safe, person centred patient care. There is some evidence of the effect of teamwork initiatives on knowledge or behaviour in specific clinical specialities, single services or single professions, but limited research is available regarding the effects of teamwork programs across multiple professions, including allied health professions, and on patient and organisational outcomes. What does this paper add? This paper describes the effect of a large-scale teamwork program implemented across multiple professions, including enablers and barriers. It presents outcomes at all four levels of Kirkpatrick's evaluation model, including the less studied behaviour and results levels. What are the implications for practitioners? This paper supports health service leaders to consider developing and implementing interprofessional teamwork programs to foster essential teamwork capabilities. Learning together about teamwork, across professional silos, will lead to collaborative, patient-centred care, which leads to safe, quality patient outcomes.

Evaluation of outcomes of continuing professional education in implant dentistry using Kirkpatrick's four-level evaluation model

Objectives: This study evaluated the outcomes of continuing professional education in implant dentistry using Kirkpatrick's four-level evaluation model. Material and methods: The study was carried out through a questionnaire distributed to dentists who attended a continuing professional education in implant dentistry, 2008~2012. They were asked to fill out questionnaires at least 6 months after its completion. Results: Mean ages of total 23 dentist was 44.8°ae8.2. Mean period after completion of education was 2.7°ae1.2. Knowledge (level 2) (r=0.71, p<0.01) and behavior (level 3) (r=0.68, p<0.01) was positively correlated with topic and methods of education in reaction (level 1). Behavior was positively correlated with knowledge (r=0.79, p<0.01). Result (level 4) was positively correlated with knowledge (r=0.64, p<0.01) and behavior (r=0.86, p<0.01). Conclusion: Reaction affects on knowledge, behavior and result in order.

A One‐Day Dental Faculty Workshop in Writing Multiple‐Choice Questions: An Impact Evaluation

Long training workshops on the writing of exam questions have been shown to be effective; however, the effectiveness of short workshops needs to be demonstrated. The aim of this study was to evaluate the impact of a one-day, seven-hour faculty development workshop at the College of Dentistry, King Saud University, Saudi Arabia, on the quality of multiple-choice questions (MCQs). Kirkpatrick's four-level evaluation model was used. Participants' satisfaction (Kirkpatrick's Level 1) was evaluated with a post-workshop questionnaire. A quasi-experimental, randomized separate sample, pretest-posttest design was used to assess the learning effect (Kirkpatrick's Level 2). To evaluate transfer of learning to practice (Kirkpatrick's Level 3), MCQs created by ten faculty members as a result of the training were assessed. To assess Kirkpatrick's Level 4 regarding institutional change, interviews with three key leaders of the school were conducted, coded, and analyzed. A total of 72 course directors were invited to and attended some part of the workshop; all 52 who attended the entire workshop completed the satisfaction form; and 22 of the 36 participants in the experimental group completed the posttest. The results showed that all 52 participants were highly satisfied with the workshop, and significant positive changes were found in the faculty members' knowledge and the quality of their MCQs with effect sizes of 0.7 and 0.28, respectively. At the institutional level, the interviews demonstrated positive structural changes in the school's assessment system. Overall, this one-day item-writing faculty workshop resulted in positive changes at all four of Kirkpatrick's levels; these effects suggest that even a short training session can improve a dental school's assessment of its students.

Building Evaluation Capability in the Public Health Workforce

Building evaluation capability and capacity is a complex task involving enhancing and building technical skills and working closely with individuals and groups to ensure evaluation learning is embedded in organisational philosophy and practice. In this article, we present the evaluation findings of the ‘Easy Evaluation’ initiative, which provides evaluation training workshops and organisational coaching and support, in order to address our central question: Are evaluation training workshops effective and what else is needed? The findings are organised in line with three levels of Kirkpatrick's four-level evaluation model: trainee reactions, learning and behaviour. The ‘reaction’ and ‘learning’ data indicate that workshops were found to be effective in developing positive responses and enhancing trainee learning for most participants. Evaluation data relating to ‘behaviour’ indicated that just over half the training participants, at least three months following the training, were implementing evaluation activities. In summary, to build sustainable evaluation practice within organisations, training needs to be supplemented with other evaluation capacity-building strategies.

Enhancing reflective practice through online learning: impact on clinical practice

PurposeTraditionally, radiographers and radiation therapists function in a workplace environment that is protocol-driven with limited functional autonomy. The workplace promotes a culture of conformity and discourages practitioners from reflective and critical thinking, essential attributes for continuing learning and advancing workplace practices. As part of the first author’s doctoral study, a continuing professional development (CPD) educational framework was used to design and implement an online module for radiation therapists’ CPD activities. The study aimed to determine if it is possible to enhance healthcare practitioners’ reflective practice via online learning and to establish the impact of reflective learning on clinical practice.Materials and methodsThe objectives of the online module were to increase radiation therapists’ knowledge in planning for radiation therapy for the breast by assisting them engage in reflective practice. The cyclical process of action research was used to pilot the module twice with two groups of volunteer radiation therapists (twenty-six participants) from Australia, New Zealand and Canada.ResultsThe online module was evaluated using Kirkpatrick’s four-level evaluation model. Evidence indicated that participants were empowered as a result of participation in the module. They began reflecting in the workplace while assuming a more proactive role and increased clinical responsibilities, engaged colleagues in collaborative reflections and adopted evidence-based approaches in advancing clinical practices.ConclusionThe study shows that it is possible to assist practitioners engage in reflective practice using an online CPD educational framework. Participants were able to apply the reflective learning they had developed in their workplace. As a result of their learning, they felt empowered to continue to effect changes in their workplace beyond the cessation of the online module.

Comprehensive evaluation of an online tobacco control continuing education course in Canada

To respond to the increasing need to build capacity for planning, implementing, and supporting tobacco control strategies, an evidence-based, online continuing education (CE) course aimed at Canadian public health professionals was developed. The purpose of this study was to comprehensively evaluate the course, Tobacco and Public Health: From Theory to Practice (http://tobaccocourse.otru.org). Rossett and McDonald's revision of Kirkpatrick's four-level evaluation model for training programs guided the evaluation design. A pre-, post-, and follow-up single group design assessed immediate reactions to course modules, knowledge change and retention, practice change, and overall perceived value of the course. Six external peer reviewers evaluated course module content. Fifty-nine participants completed all three course modules and the final online questionnaire at time 3, representing a response rate of 78%. Significant knowledge gains occurred between times 1 and 2 (p < 0.001). Although time 3 scores remained higher than time 1 scores for each module (p < 0.001), they decreased significantly between times 2 and 3 (p < 0.001). The majority of participants (93%) felt the topics covered were useful to their daily work. All but one participant felt the course was a good investment of their time, and nearly all participants (97%) stated they would recommend the course to others. Peer reviewers found that module content flowed well and was comprehensive. This comprehensive evaluation was valuable both for assessing whether course goals were achieved and for identifying areas for course improvement. We expect this design would be a useful model to evaluate other online continuing education courses.