Kirkpatrick's Evaluation Framework Research Articles

Building on strengths to develop the leadership of social workers

ABSTRACT The importance of leadership to creating supportive organizational cultures and positive team cultures for social work practice is well recognized. Yet, there are few opportunities provided for developing leadership behaviors among social workers. Jointly developed by a university and an improvement body, this paper examines the experiences of social workers participating in a post-qualifying leadership program that was aimed at developing participants’ leadership that is embedded in the principles of strengths-based practice. Employing a case study approach, the study presents findings from an end-of-program survey, semi-structured interviews, self-reflection forms, and focus groups. Following the Kirkpatrick evaluation framework as our analytical tool for data collection and analysis, our findings suggest that the design and delivery of the program provided theory informed opportunities for reflection, improving the knowledge, skills, and leadership behaviors of participants. By offering theoretical and practical opportunities for reflection, participants were more confident in claiming a leadership identity and recognizing how to reflect their social work values within their leadership. Greater emphasis in social work education and by professional bodies on the nature of social work leadership would further strengthen its legitimacy and contribution.

The new Foundation Programme Mental Health Curriculum: foundation doctors' perceptions of its importance and their competency: pre-post psychiatric placement evaluation study.

The new 2021 UK Foundation Programme Curriculum mandates foundation doctors to acquire mental health competencies. This study aimed to evaluate the effectiveness of psychiatry placements in facilitating competency attainment, foundation doctors' perceived importance of acquiring these and their preferred teaching methods. Utilising Kirkpatrick's evaluation framework, the study employed a pre-post intervention design assessing the impact of psychiatry placements on 135 foundation doctors across three cohorts from August 2021 to March 2022. Initially, foundation doctors assigned high importance to mental health competencies. Post-placements, this perceived importance improved slightly, whereas that of clinical skills scenarios slightly decreased. Significant confidence increases were observed in recognising and assessing specific psychiatric disorders. Foundation doctors favoured small seminar groups and on-the-job ad hoc teaching. Qualitative insights underscored the need for context-specific teaching. Psychiatry placements enhance foundation doctors' confidence and perceived importance of mental health competencies as specified by the curriculum. Addressing clinical scenario gaps through context-specific teaching and transferable skills development is essential. Customised teaching approaches, especially small seminars and ad hoc teaching, hold promise for effective mental health training.

Effectiveness of a modified Balint group process on empathy and psychological skills employing Kirkpatrick's evaluation framework.

To evaluate, using Kirkpatrick's evaluation model, a modified Balint group (BG) by adding 5-10 min at the end of the session, where the facilitators will debrief the residents about critical themes mentioned in the session. A quasi-experimental study with a mixed-method design was conducted among family medicine residents over 1 year, using focus groups and validated tools filled by residents and their corresponding patients. The residents' empathy through self and patient evaluation, psychological skills, and satisfaction with the educational activity were measured. The focus group revealed that the residents were aware of the change and considered it a closure to the encounter, helping decrease some uncertainty. Most of the residents did not consider the change helpful. Using validated instruments, BG was ineffective at improving the residents' empathy and psychological skills. There was a statistically significant main effect of time on Psychological Medicine Inventory (PMI) scores, F (1,13) = 7.709, p = 0.016. Debriefing at the end of BG by the facilitators about key themes may help give the residents closure, decrease the uncertainty, and make them more aware of their feelings. Nevertheless, Balint groups are still not very well accepted by the residents, and they prefer direct feedback and support groups.

Project Evaluation for Expand the Potential of the Teacher Network Award from the Princess Maha Chakri Award (PMCA) Foundation Towards the Development of Underprivileged Children in Thailand

This study aimed to evaluate the impact of a project aiming to enhance the empowerment of disadvantaged children and youth in Thailand, with a view to promoting educational equality in the country. The Kirkpatrick evaluation framework, comprising four dimensions - response, learning, behavior, and results - was applied. Data was collected from 93 teachers out of a total of 121 (76.85%) who participated in the project, as well as 28 teachers who were interviewed. The research tools included a 5-point Likert scale questionnaire comprising 26 items, and a 6-item interview guide. Statistical analysis using mean, standard deviation, and t-test was conducted to analyze the data. The results indicated that the project had a significant impact on teacher response and learning, as well as a positive impact on behavior and results. Implications for future research and practical applications are discussed.The research findings indicate that the overall assessment of the project is highly satisfactory (= 4.17). Upon consideration of the individual domains, it was found that all domains were highly satisfactory, with teachers involved in the project expressing the highest level of satisfaction with the outcomes of the project's implementation (= 4.27). In particular, the domain of behavior in the application of knowledge in practice was highly effective (= 4.22). The level of response that individuals have to the environmental sustainability project and the feelings they have towards the project (= 4.13) and the learning aspect, including understanding the principles and theories, and the skills necessary to achieve the goals of the project (= 4.06). A significant recommendation is the development of an information technology system to collect and disseminate the innovative work of teachers in order to expand its impact. It is also important to link the dissemination system of the project with capable agencies in the field.

Impact of an Observation Medicine Educational Intervention on Residents' Confidence, Knowledge, and Attitudes: A Quasi-Experimental Study.

Driven by innovations in healthcare, observation medicine (OM) is expanding as a medical specialty. Despite exponential growth, education on OM remains underemphasized in the internal medicine (IM) residency programs. We assessed the impact of an educational intervention pairing didactic and experiential learning with an interdepartmental approach on IM residents' confidence, knowledge, and attitudes when providing observation care to patients with neuro-cardiovascular diseases in the hospital setting. Our multifaceted intervention incorporated OM's principles and practice in a flipped classroom with the team-, case-, lecture- and evidence-based learning model. Kirkpatrick's evaluation model was used to assess the educational intervention's effectiveness according to the first three levels, ie, reaction, learning, and behavior, using quantitative surveys. The surveys were completed pre-intervention, and immediately upon completion of the educational intervention. Of 55 eligible residents, 55 (100%) participated in this intervention. Fifty (90%) completed the pre-intervention survey, and 21 (38%) completed the immediate post-intervention survey. Kirkpatrick's evaluation framework showed that the intervention had a positive impact on residents' motivational reaction (attention, relevance, confidence, and satisfaction [ARCS], M = 3.8, SD = 0.87), their knowledge of common observation diagnoses (pre = 49%, post = 63%), particularly on cardiac diagnostic workup and approach to patients with transient neurological symptoms (P < .05), and their behavior and self-assessment of core competency domains (pre-mean = 2.69, post-mean = 3.18, P < .001). Our multimodal intervention provides a framework for a structured OM educational experience that can be incorporated into residency training, even without a formal observation unit rotation. The analysis also offers literary data on the current state of OM education in an IM residency program and supports the need to expand OM's educational resources to counteract the growth in hospital observation services. Future research should include an analysis of residents' knowledge and skills from a longitudinal OM experience and advancing the results to residency programs where observation care is as applicable as ours.

The Role of Evaluation Methods in Health-Related E-learning: A Rapid Review

Training and development programs are increasingly delivered online with numerous studies reporting no differences in learning outcomes between online and traditional learning. However, there are no established standardized methods to evaluate the effectiveness of online learning. This review aims to map the state of research around health-related education to determine what e-learning evaluation methods are being used, the strengths or deficiencies of these methods, and which are appropriate for measuring the effectiveness of online education. Databases searched were PubMed, ProQuest, Education Resources Information Centre, Cumulative Index to Nursing and Allied Health Literature, Scopus, PsychInfo, and Medline. Studies were included if they were published between 2011 and 2021, reported health-related online education and included an evaluation component. Thirty studies were obtained from numerous countries with varied methodologies and designs. Participants ranged from undergraduate students to medical professionals. Evaluation methods included student participation, students’ reaction to the training program, self-efficacy, knowledge assessment, long-term performance, and the Kirkpatrick Evaluation Framework. The review identified that course evaluations, such as measuring student satisfaction scores alone, are insufficient when used to quantify learning effectiveness for online education. This was particularly important as studies are reporting these single metrics as positive effects of training interventions without justification. Suggestions within the reviewed papers were to adopt and implement an appropriate validated method within the course curriculum to evaluate learning outcomes.

Building the Nigerian Palliative Care Workforce: An Interdisciplinary Distance Learning Training Program.

Education and capacity building in palliative care are greatly needed in Nigeria. Currently, two institutions integrate palliative care into the undergraduate medical curriculum and no post graduate training exists. A team from the University of Lagos in Nigeria and Northwestern University in the US collaborated to design, implement, and evaluate a 12-hour virtual palliative care training program for Nigerian health professionals. This study investigated the impact of the first session of the training program on healthcare professionals' knowledge, skills, attitudes, and confidence in palliative care. The Education in Palliative and End-of-Life (EPEC) curriculum and the Kenya Hospices and Palliative Care Association (KEHPCA) curriculum were used as foundations for the program and adapted for the Nigerian context. Delivered online, the training focused on goals of palliative care, whole patient assessment, communication skills, pain management, psychosocial issues, palliative care in COVID, oncology, and HIV. A mixed-methods evaluation based on Kirkpatrick's evaluation framework was used and data were gathered from surveys and focus groups. Thirty-five health professionals completed the training. The training had a positive impact on knowledge, skills, and attitudes. Confidence in providing end-of-life care increased from 27.3% to 92.9% while confidence in prescribing medication to relieve symptoms at the end of life increased from 42.9% to 92.0%. Performance on multiple-choice knowledge tests increased by 10% (p < 0.01). All participants stated that they would recommend the program to a peer while 96.4% reported the program was relevant to the Nigerian context. Qualitative analysis suggested that the training would help participants provide more holistic care for patients, communicate better, and change how they interacted with families. Topics to be addressed in future training were identified. This virtual training can be an important element in palliative care capacity building in Nigeria and represents a model for global health collaboration.

Surgical data strengthening in Ethiopia: results of a Kirkpatrick framework evaluation of a data quality intervention

ABSTRACT Background: One key challenge in improving surgical care in resource-limited settings is the lack of high-quality and informative data. In Ethiopia, the Safe Surgery 2020 (SS2020) project developed surgical key performance indicators (KPIs) to evaluate surgical care within the country. New data collection methods were developed and piloted in 10 SS2020 intervention hospitals in the Amhara and Tigray regions of Ethiopia. Objective: To assess the feasibility of collecting and reporting new surgical indicators and measure the impact of a surgical Data Quality Intervention (DQI) in rural Ethiopian hospitals. Methods: An 8-week DQI was implemented to roll-out new data collection tools in SS2020 hospitals. The Kirkpatrick Method, a widely used mixed-method evaluation framework for training programs, was used to assess the impact of the DQI. Feedback surveys and focus groups at various timepoints evaluated the impact of the intervention on surgical data quality, the feasibility of a new data collection system, and the potential for national scale-up. Results: Results of the evaluation are largely positive and promising. DQI participants reported knowledge gain, behavior change, and improved surgical data quality, as well as greater teamwork, communication, leadership, and accountability among surgical staff. Barriers remained in collection of high-quality data, such as lack of adequate human resources and electronic data reporting infrastructure. Conclusions: Study results are largely positive and make evident that surgical data capture is feasible in low-resource settings and warrants more investment in global surgery efforts. This type of training and mentorship model can be successful in changing individual behavior and institutional culture regarding surgical data collection and reporting. Use of the Kirkpatrick Framework for evaluation of a surgical DQI is an innovative contribution to literature and can be easily adapted and expanded for use within global surgery.

Enhancing Teaching Skills Through Short Courses: A Quantitative Review of Public Health Education in Sudan

This evaluation draws evidence on the effectiveness and outcome of learning and teaching courses that were offered to a number of teaching staff of the Academies of Health Sciences, Midwifery Schools and Centres for Continuous Professional Development in Sudan. It is based on a cross-sectional study consisting of self-administered questionnaires conducted from October 2017 to January 2018. The questionnaires were used to measure the extent to which the participants of the training courses acquired knowledge and skills and applied these skills in their jobs. Data was analysed using Excel sheets and SPSS version 21 and significant tests performed. The response rate for the self-administered questionnaire was 80%. Most of participants were females, early middle-aged, bachelor’s degree holders with more than 9 years of experience in the area of teaching health care professionals. The results are presented following the Kirkpatrick framework for evaluation of training, which was used to measure participants' reactions, learning and change of behaviour. Most of the participants were satisfied with the design and content of the course but expressed concerns regarding its duration and lack of residential facilities for course participants. Pre and post tests were used to measure learning. Comparison of the two results showed that participants gained knowledge in writing learning outcomes, although presentation skills showed no improvement following the training and the use of technology remained a challenge even after the course. In terms of application of the newly gained knowledge and skills, the findings illustrated that over 90% of the participants were satisfied with what they had learned after 3 months or more following the training course, and over 65% of the participants stated that the knowledge gained from the training remained useful. In conclusion, short courses on teaching and learning can be a valuable investment to both instructors and students. The evaluation has shown that course participants were satisfied with the training courses, gained new knowledge, and were able to apply the acquired knowledge and skills without any difficulties. It is therefore suggested to cascade these training courses to all remaining States in Sudan in order to improve the skills and capabilities of teaching staff.

Antenatal Point-of-Care Ultrasound (PoCUS) in rural and remote Australian communities: an evaluation of training

Introduction Access to health services, including antenatal ultrasound scanning, is a key risk factor associated with neonatal and maternal mortality and morbidity. Proportionately fewer pregnant women in rural and remote Australia have access to timely ultrasound services compared to women in metropolitan and regional areas, due in part to a chronic shortage of trained sonographers1,2,3. PoCUS is receiving growing attention and offers a practical solution to address this deficit in best practice care. However, questions remain surrounding how to best teach PoCUS and ensure training is contextually delivered to our remotely located clinicians who face considerable barriers to professional development. Objective This research aimed to deliver PoCUS training to small groups of GPs and midwives recruited from rural and remote areas in Australia. The effectiveness of the teaching and experiences of the trainees were examined and the impact of the training on clinical practice explored. Methods Two intensive two-day antenatal PoCUS training workshops (endorsed by ASUM and ISUOG) were conducted for twelve GPs and thirteen midwives in early 2018. They consisted of lectures and practical hands-on sessions with high fidelity simulators and live pregnant models. Pre- and post-training assessment and evaluation was conducted and progress surveyed at 3 and 6 months. The Kirkpatrick Evaluation Framework (KEF) was used to appraise the efficacy of the PoCUS training provided. Results All participants found the training useful and relevant to their role, and highly rated the workshop design, content and delivery. The advantages of High fidelity simulators and live pregnant volunteers featured heavily in the feedback. Pre/post testing analysis showed 26% improvement in antenatal US knowledge (P Conclusion PoCUS presents a viable solution to assist resource-poor settings in rural and remote Australia. Intensive workshops are able to equip clinicians with valuable life-saving skills which can benefit our most disadvantaged communities. Geographical isolation and lack of onsite specialist supervision poses an ongoing challenge to the development of remote trainees’ PoCUS skills.

Effectiveness of Online Cancer Education for Nurses and Allied Health Professionals; a Systematic Review Using Kirkpatrick Evaluation Framework.

Embedding online learning within higher education can provide engaging, cost-effective, interactive and flexible education. By evaluating the impact, outcomes and pedagogical influence of online cancer and education, future curricula can be shaped and delivered by higher education providers to better meet learner, health care provider and educational commissioners' requirements for enhanced patient care and service delivery needs. Using the Kirkpatrick's four-level model of educational evaluation, a systematic review of the effectiveness of online cancer education for nurses and allied health professionals was conducted. From 101 articles, 30 papers were included in the review. Educational theory is not always employed. There is an absence of longitudinal studies to examine impact; an absence of reliability and/or validity testing of measures, limited experimental designs taking account of power and few attempts to mitigate bias. There is, however, an emerging innovative use of mobile/spaced learning techniques. Evidence for clinical and educational effectiveness is weak offering insights into experiences and participant perceptions rather than concrete quantitative data and patient-reported outcomes. More pedagogical research is merited to inform effective evaluation of online cancer education, which incorporates and demonstrates a longer-term impact.

Assessing a Media Production Workshop based on Kirkpatrick Evaluation Model

Background and Purpose: There are 3 specific reasons for education evaluation: to justify the existence and budget of the training department by showing how it contributes to the organization’s objectives and goals, to decide whether to continue or discontinue training programs, and to gain information on how to improve future training programs. We aimed to assess the effectiveness a media production workshop during 2014. Methods: In this quasi-experimental study a workshop related to producing media was evaluated among the health department staff of Shahid Beheshti University of Medical Sciences based on Kirkpatrick's model. Data were collected through DREEM questionnaire and analyzed using SPSS software, version 17. Results: In the reaction level, the mean±SD obtained from the learner’s perception of educational environment was 144.92±22.53. The mean±SD scores of learner’s perception of learning, learner’s perception of course teachers, learner’s academic self-perception, learner’s perceptions of atmosphere, and learner’s social self perceptions were 29.61±4.87, 33.61±4.78, 21.84±3.71, 36.38±7.07, and 20.15±3.13, respectively. Conclusions: In the reaction level, the correlation between the scales of the learner’s perception of educational environment showed that better learning can be provided with minimal changes in each of the scales. The results of correlation between the reaction to the learning level showed that the learner’s perception of teacher compared with other scales has an important role in raising awareness. Kirkpatrick evaluation framework, confirms the success of the workshop at the level of reaction and learning. Future studies using case and control groups are recommended to achieve better results. Keywords: Kirkpatrick evaluation model, DREEM, Workshop, Media production

Crew resource management training in the intensive care unit. A multisite controlled before–after study

IntroductionThere is a growing awareness today that adverse events in the intensive care unit (ICU) are more often caused by problems related to non-technical skills than by a lack of...

Teaching Assistant Professional Development in Biology: Designed for and Driven by Multidimensional Data.

Graduate teaching assistants (TAs) are increasingly responsible for instruction in undergraduate science, technology, engineering, and mathematics (STEM) courses. Various professional development (PD) programs have been developed and implemented to prepare TAs for this role, but data about effectiveness are lacking and are derived almost exclusively from self-reported surveys. In this study, we describe the design of a reformed PD (RPD) model and apply Kirkpatrick's Evaluation Framework to evaluate multiple outcomes of TA PD before, during, and after implementing RPD. This framework allows evaluation that includes both direct measures and self-reported data. In RPD, TAs created and aligned learning objectives and assessments and incorporated more learner-centered instructional practices in their teaching. However, these data are inconsistent with TAs' self-reported perceptions about RPD and suggest that single measures are insufficient to evaluate TA PD programs.

Qualitative evaluation of interprofessional learning initiatives in practice: application of the contact hypothesis

Objectives - The evaluation aimed to explore reactions to, learning from and the perceived impact and sustainability of practice-based interprofessional learning initiatives established by The Trent Universities Interprofessional Learning in Practice project. Methods - The evaluation had a qualitative design and was guided by Kirkpatrick's evaluation framework. Facilitators, managers, practitioners, students and service users were invited to participate. This article focuses upon three of the project's practice-based interprofessional initiatives. Fifteen participants were interviewed about their experiences of, and opinions about, the initiative in their practice setting. A thematic analysis was conducted, following which findings were considered through the lens of the 'contact hypothesis' a theoretical framework which sets out variables purported to influence the success of inter-group contact. Results - Seven themes were identified. Four derived from Kirkpatrick's framework: reaction, learning, impact and sustainability. Initial reactions were positive; however learning and impact were considered minimal, and sustainability unlikely. The evaluation revealed significant problems in developing successful, sustainable, service user-focused interprofessional learning opportunities in these three sites. Three final themes were based upon contact hypothesis variables which helped elucidate the disappointing outcomes: organisational support, positive expectations, and co-operation/working together. Conclusions - Results offer insights which could help medical education establishments and their practice partners develop successful practice-based interprofessional learning initiatives for students in the future. Initiatives should be designed to meet local working practices and opportunities; there should be an awareness that change is challenging and time-consuming; and preparation of and support in host organisations at all levels should be a priority.

Effectiveness of classroom based crew resource management training in the intensive care unit: study design of a controlled trial

BackgroundCrew resource management (CRM) has the potential to enhance patient safety in intensive care units (ICU) by improving the use of non-technical skills. However, CRM evaluation studies in health care are inconclusive with regard to the effect of this training on behaviour and organizational outcomes, due to weak study designs and the scarce use of direct observations. Therefore, the aim of this study is to determine the effectiveness and cost-effectiveness of CRM training on attitude, behaviour and organization after one year, using a multi-method approach and matched control units. The purpose of the present article is to describe the study protocol and the underlying choices of this evaluation study of CRM in the ICU in detail.Methods/DesignSix ICUs participated in a paired controlled trial, with one pre-test and two post test measurements (respectively three months and one year after the training). Three ICUs were trained and compared to matched control ICUs. The 2-day classroom-based training was delivered to multidisciplinary groups. Typical CRM topics on the individual, team and organizational level were discussed, such as situational awareness, leadership and communication. All levels of Kirkpatrick's evaluation framework (reaction, learning, behaviour and organisation) were assessed using questionnaires, direct observations, interviews and routine ICU administration data.DiscussionIt is expected that the CRM training acts as a generic intervention that stimulates specific interventions. Besides effectiveness and cost-effectiveness, the assessment of the barriers and facilitators will provide insight in the implementation process of CRM.Trial registrationNetherlands Trial Register (NTR): NTR1976

Providing demonstrable return‐on‐investment for organisational learning and training

PurposeThe aim of this paper is to present a holistic approach to training, that clearly demonstrates cost savings with improved effectiveness and efficiencies that are aligned to business objectives.Design/methodology/approachExtending Kirkpatrick's evaluation framework with Phillips's return‐on‐investment (ROI) concepts, the paper conveys a number of successes; including trainee satisfaction and the capturing of improved knowledge and skills.FindingsThe paper includes case studies of how, and to what extent this knowledge has been applied with examples of resulting efficiency savings. The paper shows that there is growing agreement that one of the primary drivers, if not the key driver of long‐term organisational effectiveness, is the ability of an organisation to learn effectively. The methodology requires some additional assessment and course preparation to establish a basis from which to demonstrate learning effectiveness. The financial benefits of the applied learning are far greater than the additional preparatory costs.Practical implicationsLearning organisations that anticipate, react to change and learn, are likely to maintain a competitive advantage. These organisations are constantly looking for more effective and efficient ways of training. Paradoxically, other organisations will often slash training budgets in times of hardship, as training departments are unable to demonstrate the effectiveness of their programmes.Originality/valueThe paper presents a practical example of how training should be applied to truly demonstrate its value in the workplace

Effect of crew resource management training in a multidisciplinary obstetrical setting

To assess the effect of a Crew Resource Management (CRM) intervention specifically designed to improve teamwork and communication skills in a multidisciplinary obstetrical setting. Design--A before-and-after cross-sectional study designed to assess participants' satisfaction, learning and change in behaviour, according to Kirkpatrick's evaluation framework for training programmes. Setting--Labour and delivery units of a large university-affiliated hospital. Participants--Two hundred and thirty nine midwives, nurses, physicians and technicians from the department of anaesthesia, obstetrics and paediatrics. Intervention--All participants took part in a CRM-based training programme specifically designed to improve teamwork and communication skills. Principal measures of outcome-We assessed participants' satisfaction by means of a 10-item standardized questionnaire. A 36-item survey was administered before and after the course to assess participants' learning. Behavioural change was assessed by a 57-item safety attitude questionnaire measuring staff's change in attitude to safety over 1 year of programme implementation. Most participants valued the experience highly and 63-90% rated their level of satisfaction as being very high. Except for seven items, the 36-item survey testing participants' learning demonstrated a significant change (P<0.05) towards better knowledge of teamwork and shared decision making after the training programme. Over the year of observation, there was a positive change in the team and safety climate in the hospital [odds ratio (OR) 2.9, 95% confidence interval (CI) (1.3-6.3) to OR 4.7, 95% CI (1.2-17.2)]. **There was also improved stress recognition [OR 2.4, 95% CI (1.2-4.8) to OR 3.0, 95% CI (1.0-8.8)]. The implementation of a training programme based on CRM in a multidisciplinary obstetrical setting is well accepted and contributes to a significant improvement in interprofessional teamwork.