Kirkpatrick's Evaluation Model Research Articles

Evaluasi Program Pelatihan Guru di Pusdiklat Tenaga Teknis Pendidikan dan Keagamaan Kementerian Agama Republik Indonesia

This study aims to evaluate the Teacher Training Program at Training Center for Education and Religious Technical Personnell. The basic problem is that there is no post-training evaluation as an integral part of the evaluation of training implementation. The evaluation model used in this study is the Kirkpatrick evaluation model which is reaction level, learning level, behavior level and result level to get a more comprehensive picture of the Teacher Training Program. This study uses a qualitative approach with descriptive methods. Data was conduct using questionnaire, interview, and document studies. The results of this study indicate that on reaction level, based on the results of the achievement analysis are in the high category, learning level of the program which are also in the high category, behavior level are in the high category, however result level of the program are in the moderate category. The findings of the evaluation of this result prove the importance of the training center conducting post-training evaluations to determine the impact of the training on the performance of participants and educational institutions. This research can be used as input for the Training Center to be consistent in evaluating both during training and after training as stated in Kirkpatrick's theory.

The Effect of Global Surgery Fellowship for Tertiary Hospital Professionals from Myanmar: Current Issues and Future Challenges

As the surgical burden of diseases grows higher than ever, the development of skilled surgeons and surgery teams is of fundamental importance. However, there is scarce evidence of the effectiveness of surgical training programs. Our study aims to evaluate the effectiveness of a global surgery fellowship program. In 2018, Myanmar subspecialty surgeons were trained for a 3-month in tertiary hospital of Korea. We evaluated the reactions, learning, and transfer of the 11 trainees using Kirkpatrick's Evaluation Model. Myanmar and Korean tertiary hospital. Eleven participants of fellowship from Myanmar. Participants' overall reaction scores were 4.45 ± 0.52 out of 5.00. Regarding overall competency, the trainees assessed themselves as at an "advanced beginner level" (3.63 ± 1.38 out of 10.00) at the beginning and at a "competent level" (5.40 ± 1.70 out of 10.00) after the training (p for difference = 0.014). The trainees stated that their transfer in clinical practice was between "quite applicable" and "I am very sure that I can do this" (3.39 ± 0.61 out of 4.00). Their transfer in education was second, ranked around "quite applicable" (3.21 ± 0.77 out of 4.00), and their transfer in research was last and ranked "applicable, but just a little" (2.81 ± 0.67 out of 4.00). However, in our online survey, native and foreign trainers had mixed opinions about the transfer of training and were not entirely confident in the trainees' competency. Trainees felt that there were multifactorial challenges for transfer and cited infrastructure, facilities, human resources, perioperative care, and financing. Surgical trainees reported high-level satisfaction and increased knowledge after training. However, further onsite surgical experiences are necessary for a successful transfer to the real setting. Challenges for such transfers were multifactorial, and surgical care must be systemically strengthened.

Fostering effective use of ICT in agricultural extension: participant responses to an inaugural technology stewardship training program in Trinidad

ABSTRACT Purpose: This study sought to investigate agricultural extension practitioners’ response to an in-service training program offered by the University of the West Indies intended to foster effective use of ICTs (information and communication technologies) through a technology stewardship approach. Design/Methodology/Approach: This study reports on participants responses to an in-service training course on technology stewardship. Data were collected using a pre-course survey, formal course evaluation, classroom observation, and semi-structured interviews with participants. Kirkpatrick's four-level evaluation model was used to structure analysis of the results. Findings: Findings from this study show high levels of receptivity to the technology stewardship training course, that learning objectives of the course are achievable when offered as an in-service training program, and that most participants indicated an intent to apply their learning in a post-course activity. Results from the study raise a number of considerations for future course design and institutional policy in order to better support technology stewardship in practice. Practical implications: Technology stewardship training shows promise as a leadership-oriented approach for expanding the effective use of ICTs among communities of practice for agricultural extension in the Caribbean and elsewhere. Theoretical implications: This study applies key principles of social learning theory from a communities of practice approach with theoretical implications for the design of in-service training related to ICT adoption and use for agricultural extension education and policy. Originality/value: This study reports on the first known application of the technology stewardship model from the communities of practice literature to agricultural extension in the Caribbean.

Micro-feedback skills workshop impacts perceptions and practices of doctoral faculty

BackgroundDoctoral supervision is a distinct form of supervision with clearly defined responsibilities. One of these is the delivery of effective face-to-face feedback to allow supervisees to improve upon their performances. Unfortunately, doctoral supervisors, especially of health sciences, are often not trained in supervisory skills and therefore practice mostly on a trial and error basis. Lack of understanding of the feedback process leads to incongruence in how supervisors and supervisees perceive feedback. However, standardized training practices like microteaching can allow supervisors to acquire effective feedback practices. In this study we employed a schematic approach of microteaching, that is micro-feedback, in a workshop to develop feedback skills of doctoral supervisors, and assessed the overall effectiveness of this training using the Kirkpatrick evaluation framework.MethodologyThis was a Quasi-experimental study with a repeated measures and a two-group separate sample pre-post test design. A micro-feedback skills workshop was organized to enhance feedback skills of doctoral supervisors using microteaching technique. The first two levels of the Kirkpatrick evaluation model were used to determine the workshop’s effectiveness. An informal Objective Structured Teaching Exercise (OSTE) was used to assess feedback skills of the supervisors, both before and after the workshop. A questionnaire was developed to compare pre-and post-workshop perceptions of the supervisors (n = 17) and their corresponding supervisees (n = 34) regarding the ongoing feedback practice.ResultsDespite the hectic schedule, most doctoral supervisors (17 of 24, 71%) were willing to undertake faculty development training. Participants indicated a high level of satisfaction with the workshop. A learning gain of 56% was observed on pre-post OSTE scores. Prior to the workshop, perceptions of how supervisees should be given the feedback differed significantly between supervisors and supervisees with an effect size difference of r = 0.30. After the workshop there was a negligible difference in perceptions between supervisors and supervisees (r = .001). Interestingly, supervisors shifted their perceptions more toward those that were originally held by the supervisees.ConclusionsThese findings suggest that well-designed and properly assessed structured programs such as micro-feedback workshops can improve how doctoral supervisors provide feedback to their supervisees and align supervisors’ perceptions of that feedback with those of their supervisees.

Using the Kirkpatrick Model to evaluate the Maternity and Neonatal Emergencies (MANE) programme: Background and study protocol

IntroductionOver 310 000 women gave birth in Australia in 2016, with approximately 80 000 births in the state of Victoria. While most of these births occur in metropolitan Melbourne and...

Planning, construction and use of handmade simulators to enhance the teaching and learning in Obstetrics.

Objective:to describe the development process and present the results of a pilot study on the use of low-cost handmade simulators for teaching and learning Obstetrics.Method:presentation of 3 low-cost simulators designing, based on educational needs identified in real-world training contexts. The developing process is presented in detail and each simulator was tested and re-tested, being submitted to improvements until their final version. The simulators presented are: delivery simulator shorts, Neoprene uterus for postpartum hemorrhage management, and perineal repair simulator. A pilot study was carried out to evaluate the perception of apprentices through a structured questionnaire, using the Kirkpatrick evaluation model. Data were descriptively analyzed.Results:the respondents (31 apprentices) positively evaluated the simulators, perceiving significant gains in theoretical knowledge, ability to solve clinical problems and decreased anxiety to deal with situations similar to those simulated.Conclusion:low-cost, handmade simulators are feasible and effective, resulting in positive learner evaluations. Their availability as open technology allows the dissemination of their use.

Evaluation of a training program on primary eye care for an Accredited Social Health Activist (ASHA) in an urban district.

Purpose:An Accredited Social Health Activist (ASHA) available in community could be a potential primary eye care (PEC) worker. Training programme for ASHAs on PEC was undertaken & evaluated in a district of a capital city.Methods:ASHAs selected randomly from a district were imparted one day training on PEC & expected to refer patients to nearby Vision Centres (VC). Their knowledge was assessed before & after training and re-evaluated 1 year later. ASHAs were asked to conduct vision screening of 40+ population in their areas and ASHA referrals were noted by Optometrist in VC. Focus Group Discussions (FGD) of ASHAs were held to find barriers & facilitating factors in engaging ASHAs in PEC. Training was evaluated using Kirkpatrick's evaluation model for measuring reactions, learning, behaviour and results.Results:Mean knowledge score increased from 14.96 (±4.34) pre-training to 25.38 (±3.48) post- training and sustained at 21.75 (±4.16) at 1year. Monthly average OPD of vision centres increased by 23.6% after ASHA training. FGDs revealed that ASHAs were willing to work in eye care for awareness generation and patient facilitation but were hesitant in conducting vision screening.Conclusion:ASHAs can be trained as PEC workers provided they have adequate support.

Evaluation of a mentored student project programme using reflective summaries.

. Undergraduate research experience has become increasingly relevant for today's medical students, considering the professional requirements of their challenging future. . In the mentored student project (MSP) programme at Melaka Manipal Medical College, students undertake a short-term group research project under the guidance of their mentor. After data collection and analysis, students are required to write an abstract, present a poster and also write individual reflective summaries of their research experience. We evaluated the MSP programme using reflective summaries of a batch of undergraduate medical students. Data from 41 reflective summaries were analysed using the thematic analysis approach. The learning outcomes at the third and fourth levels of the Kirkpatrick evaluation model were determined from the summaries. . Students' reflective summaries indicated that they were satisfied with the MSP experience. In all the summaries, there was a mention of an improvement in teamwork skills through MSP. Improved relations with mentors were another relevant outcome. Improvement in communication skills and a positive change related to research attitude were also reported by students. . Reflective summaries as a means to evaluate the MSP programme was found to be an easy, feasible and cost-effective method. The qualitative approach adopted for data analysis enabled the programme coordinators to assess the strengths and barriers of the programme.

Try-It-On: Preparing family nurse practitioners to use holistic integrative interventions to reduce opioid prescriptions in chronic pain management.

In the United States, more than 130 people die each day from an opioid overdose. Nonopioid chronic pain management options are necessary in primary care. This educational innovation describes a new curriculum to teach future family nurse practitioner (FNP) prescribers holistic integrative interventions to decrease overprescribing of opioids for chronic pain management. The Modeling and Role Modeling theory and Kirkpatrick Evaluation Model guided the development and implementation of the curriculum innovation using a Try-It-On teaching-learning strategy, which focuses on immersion experience. The focus was to teach students to communicate effectively with patients in an effort to increase patient awareness of the pathophysiology of pain, risks of opioid use, and holistic integrative pain management options. Students learned to manage chronic pain within a holistic focus through immersion within integrative modalities, such as yoga, meditation, mindfulness, and guided imagery. Pre and post participation evaluations documented students' increased comfort level managing patients with chronic pain. Students reported they were much more likely to prescribe holistic integrative modalities after completing the Try-It-On learning modules. Holistic integrative interventions are a viable treatment option and/or adjunct treatment for chronic pain management. In conclusion, using the Try-It-On, teaching-learning strategy provided FNP students with the tools needed to prescribe nonopioid holistic integrative interventions to manage chronic pain.

Does a Long-Term Quality and Safety Curriculum for Health Care Professionals Improve Clinical Practice? An Evaluation of Quality Improvement Projects.

Little is known about the effects on clinical practice of continuing education quality and safety curricula. The aim of this study is to gain insight into learning outcomes on the fourth level of the Kirkpatrick evaluation model for systematically deployed quality improvement projects performed by health care professionals during a Masters in Healthcare Quality and Safety in the Netherlands. The researchers reviewed 35 projects led by health care professionals in 16 different hospitals to determine their scopes and effects. Afterward, professionals took an online survey to determine the extent of their project's sustainability and spread. Improving health care safety was the most prevalent quality dimension (n = 11, 31%). A positive change was measured by professionals for 64% (n = 35) of the primary outcomes. Statistical significance was measured in 19 (35%) of the outcomes, of which nine (47%) were found to have a statistically significant effect. A minority of professionals (17%) judged their project as sustained by the department, while some stated that the intervention (37%) or the results of the project (11%) had spread. Although most projects indicated an improvement in their primary outcomes, only a few resulted in statistically significant changes. Teaching professionals in using evaluation methods that take into account the complex context where these projects are performed and teaching them leadership skills is needed to reduce the likelihood of unmeasured outcomes. Analyzing learning experiences of professionals in performing the project is important to see what they learned from performing quality improvement projects, providing experiences that may lead to sustainable effects in future projects.

Education Program Evaluation of Air Force Unit Command School (SEKKAU) In the Development of Human Resources

The purpose of this study is to obtain empirical and analytic data on substantive and important issues that affect the implementation of the Air Force Unit Command School’s education program. The study was conducted from 1 January 2019 to 30 June 2019 at Air Force Unit Command School (Sekkau) in Jakarta. This study used a qualitative approach with the CIPP model to evaluate the results and benefits of implementing education at Sekkau. Respondents in this study were Sekkau’s students who were studying at the time of data gathering, former Sekkau’s student serving in Air Force units, as well as instructors and officers who have authority on policies of applied at Sekkau. The informants in this study were Air Force Chief of Staff (Kasau), Assistant to Chief of Air Force Staff for Personnel (Aspers Kasau), Commander of Air Force Education and Training Command (Dankodiklatau), Head of Air Force Education Service (Kadisdikau), Sekkau Commander, and Sekkau alumni batch 104 of the 2017 academic year. The sampling procedure for this population used multi-stage (clustering). The sample selection for the survey was carried out using a purposive sampling method of 15 respondents distributed proportionally consisting of Kasau, Aspers Kasau, Dankodiklatau, Kadisdikau, Dansekkau, and Sekkau alumni of the batch 104 of the 2017 academic year. The selection of population and respondent samples, especially in modeling, was adjusted to environmental conditions and the number of interviewees to be taken i.e. those who were considered to be able to represent and understand the issues under study.The findings are: (1) the educational program of Sekkau is in accordance with the background or context of the program, which is seen based on aspects of needs, aspects of the problem and aspects of objectives, (2) the available support for education program at Sekkau in general is in a sufficiently supportive condition for the effectiveness of the education and training program for students which is seen based on aspects of human resources, budget, and the curriculum (3) implementation of educational programs were well done based on aspects of timeliness, application of learning methods, evaluation and monitoring, development of human resources, recruitment of teaching staff and staff and services to students, (4) the achieved results of the educational program at Sekkau in general was quite good from the aspect of responsiveness, toughness and vigour. The novelty of this research is that the evaluation of education program of Sekkau was obtained through two design models, namely using the David Stufflebeam CIPP evaluation model and the Four Models Kirkpatrick evaluation model that has never been done before. The results of program evaluation in this study were also obtained through the match between the idealism of evaluation criteria that were built with real conditions in the field that were different from previous studies. These are the novelties resulting from this research.

Strengthening teamwork capability in allied health: implementation of a team development program in a metropolitan health service.

Objective Collaborative practice is critical to optimising patient outcomes in contemporary healthcare settings. Evidence suggests interprofessional learning is an effective way to develop teamwork capabilities, yet these skills are traditionally developed in professional silos, or not at all. This study evaluated the implementation of a team development program, the Team Management Systems (TMS) program, for allied health staff within a large metropolitan health service. Methods A mixed-methods audit-quality improvement study was conducted, using Kirkpatrick's four-level evaluation model to structure evaluation of the program. Semistructured questionnaire and workforce survey data were retrieved immediately, 6 months and 1-2 years after training and applied to each level of the model (Reaction, Learning, Behaviour, Results). Results In all, 886 staff participated in the TMS program from 2014 to 2018. High satisfaction with the program was observed. Knowledge of what constitutes effective teamwork improved significantly (P=0.008) in TMS participants compared with a matched untrained cohort. Participants reported positive behaviour change and continued engagement with TMS principles 6 months after training. Perceived impact of the program on patient and/or organisational outcomes was evident, although less compelling than the changes to knowledge and behaviour. Conclusions The TMS program yielded positive effects on staff satisfaction, knowledge, team dynamics and team behaviours. These findings demonstrate the significant value of such initiatives to enhance the capability and effectiveness of interdisciplinary healthcare teams. What is known about the topic? Complex conditions, increasing comorbidities, specialisation and scarcity of resources mean healthcare workers need to work effectively in teams to achieve quality, safe, person centred patient care. There is some evidence of the effect of teamwork initiatives on knowledge or behaviour in specific clinical specialities, single services or single professions, but limited research is available regarding the effects of teamwork programs across multiple professions, including allied health professions, and on patient and organisational outcomes. What does this paper add? This paper describes the effect of a large-scale teamwork program implemented across multiple professions, including enablers and barriers. It presents outcomes at all four levels of Kirkpatrick's evaluation model, including the less studied behaviour and results levels. What are the implications for practitioners? This paper supports health service leaders to consider developing and implementing interprofessional teamwork programs to foster essential teamwork capabilities. Learning together about teamwork, across professional silos, will lead to collaborative, patient-centred care, which leads to safe, quality patient outcomes.

Fourteen years of quality improvement education in healthcare: a utilisation-focused evaluation using concept mapping

BackgroundThe need for training in quality improvement for healthcare staff is well acknowledged, but long-term outcomes of such training are hard to evaluate. Behaviour change, improved organisational performance and results...

Evaluation of a Workshop on Social Determinants of Health based on Kirkpatrrick Model

Introduction: Evaluation provides truly useful information about the effectiveness of the educational programs. Kirkpatrick model provides a four-level training evaluation framework (reaction, learning, behavior, and results). The present research aimed to investigate the effectiveness of a workshop on “social determinants of health” held for the health staff of a medical center in Bandar Abbas Province, in the South of Iran. Methods: In the present research, a training workshop was held on social determinants of health for 40 staff of a medical center. Later, this workshop was evaluated based on the Kirkpatrick training evaluation model and a researcher-constructed questionnaire. Descriptive and inferential statistics were used to analyze the data in SPSS-19. Results: The participants' mean scores of satisfaction (reaction), learning, and behavior were 41.47±4.34, 14.9±1.93, and 10.72±3.8, respectively. The overall effectiveness (reaction, learning and behavior) was evaluated as optimal. The most effective items in reaction, learning, and behavior levels were awareness-raising, better performance of tasks at work, and use of the acquired content in monitoring work, respectively. Conclusion: Staff training to provide them with job-related practical knowledge not only satisfies them, but also improves their professional learning and performance. Kirkpatrick training evaluation model needs to be applied to other staff training programs at four levels of reaction, learning, behavior, and results. Those who develop and implement training courses need to be acquainted with these constructs.

Exploration on the Backbone Training System of College Students in Ideological Propaganda Work in the New Media Age

It is of great strategic significance to cultivate the backbone of college students in propaganda work in the new media era to enhance the education of socialist core values and consolidate the ideological position of colleges and universities. Based on the operation system of ideological propaganda team and the career model of student backbone in colleges and universities, this paper explained a “T” type training system of core students in ideological propaganda work in colleges and universities, which provided scientific theoretical guidance for the selection, cultivation and management of student leaders group. In view of the actual work of student backbone group in ideological propaganda work, based on the Kirkpatrick evaluation model, the evaluation model of student cadre training effect in ideological propaganda work in colleges and universities was constructed, with a view to realizing systematic, standardized and scientific training feedback mechanism.

Measuring the effects of guided clinical reasoning on the Advanced Nursing Process quality, on nurses' knowledge and attitude: Study protocol.

AimThis article is a report of a study protocol designed to examine the effects of guided clinical reasoning on the quality of the Advanced Nursing Process—the evidence‐based version of the traditional nursing process. It aims to describe the theoretical framework—Kirkpatrick's evaluation model, the key concepts and the instruments for the planned study.DesignA complex experimental intervention study using data and method triangulation is proposed.MethodsRegistered Nurses (N = 92), nursing records (N = 180) and 24 patients will be included. Nurses’ knowledge and attitude will be evaluated by questionnaires/tests, their clinical performance by observations. Patients’ perspective will be addressed by qualitative interviews and patient records by using the instrument Quality of Diagnoses, Interventions and Outcomes revised (Q‐DIO R).DiscussionKirkpatrick's model (including quantitative and qualitative methods) is providing evaluations from different perspectives on the quality of the Advanced Nursing Process and on intervention effects.

Development and evaluation of an online course about the social accountability of medical schools.

ObjectivesThe objective of this communication is to describe the development, implementation, and evaluation of a pioneering online course about the social accountability of medical schools intended to develop the skills of academic staff at medical schools in the eastern Mediterranean region.MethodsThis four-module course was developed using a six-step approach for curriculum development. Three faculty members with vast experience in social accountability delivered the course online through the MOODLE platform to participants. The content and appropriateness of the course were evaluated using Kirkpatrick's evaluation model, by offering a self-administered questionnaire that assesses the participants' opinions and feedback besides the analysis of the responses of the participants to the discussion points.ResultsSixteen participants from four countries were admitted to the course. An overall 75% of the participants completed four modules. The mean number of online discussion threads was reported to be 36 responses per module. All participants regarded the course as having a clear take-home message. The majority agreed that the course introduced new concepts and corrected some of their misunderstandings about social accountability in medical schools. The main problems that participants experienced were time constraints and technical Internet problems.ConclusionThe application of the concept of social accountability in the day-to-day work of medical schools requires faculty to be informed and trained. The online course provides a flexible way to conduct faculty development programmes.

Capacity Building for Public Health: Participant-Guided Training.

Growth in the demand for public health services, along with limited funding, makes workforce collaboration and capacity building imperative. The faculty and staff of the Midwestern Public Health Training Center, with two Robert Wood Johnson Public Health Nurse Leaders, postulated that training could be more effective, and public health workers more effective in the field, if workers contributed to training format and content. The learning paradigm was tested on diabetes prevention and self-management programs. Public health professionals were surveyed on infrastructure, practices, roles, and gaps in diabetes-related services. Responses influenced the format and content of a one-day diabetes summit training program. Participants submitted evaluations immediately afterward. Eight months postsummit, participants were surveyed to self-assess behavioral changes attributed to the training. Using the Kirkpatrick model for evaluation, participants (n = 112) stated that the training met their expectations and that knowledge gained was consistent with stated training objectives. Qualitative postsummit survey results indicated that improvements in participants' delivery of diabetes prevention services to the public could be attributed to the training they received at the summit. Results suggest that training about specific programs and practices, as well as facilitated sessions of collaboration, can yield individual and organizational change.

Data Collection Approaches to Enable Evaluation of a Massive Open Online Course About Data Science for Continuing Education in Health Care: Case Study.

BackgroundThis study presents learner perceptions of a pilot massive open online course (MOOC).ObjectiveThe objective of this study was to explore data collection approaches to help inform future MOOC evaluations on the use of semistructured interviews and the Kirkpatrick evaluation model.MethodsA total of 191 learners joined 2 course runs of a limited trial of the MOOC. Moreover, 7 learners volunteered to be interviewed for the study. The study design drew on semistructured interviews of 2 learners transcribed and analyzed using Braun and Clark’s method for thematic coding. This limited participant set was used to identify how the Kirkpatrick evaluation model could be used to evaluate further implementations of the course at scale.ResultsThe study identified several themes that could be used for further analysis. The themes and subthemes include learner background (educational, professional, and topic significance), MOOC learning (learning achievement and MOOC application), and MOOC features (MOOC positives, MOOC negatives, and networking). There were insufficient data points to perform a Kirkpatrick evaluation.ConclusionsSemistructured interviews for MOOC evaluation can provide a valuable in-depth analysis of learners’ experience of the course. However, there must be sufficient data sources to complete a Kirkpatrick evaluation to provide for data triangulation. For example, data from precourse and postcourse surveys, quizzes, and test results could be used to improve the evaluation methodology.

Effect of role-playing on learning outcome of nursing students based on the Kirkpatrick evaluation model.

BACKGROUND:Evaluation of educational courses is important for estimating the achievement of learning goals and identifying the best way to learn. The present study is an attempt to assess the effectiveness of education through role-playing on the learning outcomes in nursing students based on the Kirkpatrick's evaluation model.MATERIALS AND METHODS:A quasi-experimental study was conducted with participation of 74 nursing students at Iran University of Medical Sciences in 2016–17. The participants were selected through census and were randomly allocated to control (n = 35) and experimental (n = 39) groups. The common method of education was implemented for the control group, and the experimental group experienced role-playing educational method. In the next semester, each student was assigned to educate two patients at the hospital. The knowledge level of the participants at the end of the semester and patients’ satisfaction with the educations by students in the next semester were measured as the outcomes of learning. Kirkpatrick's model was used to assess the learning outcomes.RESULTS:The mean score of students, at the second level of Kirkpatrick's model, in the experimental group (63.85 ± 13.88) was significantly higher than that of the control group (46.41 ± 16.22, P < 0.001). The mean score of patients’ satisfaction with patient educational performance, at the fourth level of the model, in the experimental group (73.26 ± 3.47) was significantly higher than that of the control group (47.32 ± 6.83, P < 0.001).CONCLUSIONS:The evaluation by the Kirkpatrick's model showed that use of role-playing method improved learning outcome of nursing students.