Objective Structured Assessment Of Debriefing Research Articles

Improvement in student-led debriefing analysis after simulation-based team training using a revised teamwork assessment tool

BackgroundTeam debriefing is an important teamwork development intervention for improving team outputs in healthcare. Debriefing is a key component of experiential team training teamwork development interventions such as simulation-based training. Improving the quality of debriefing of healthcare teams, therefore, has multiple benefits. We investigated whether the quality of student-led debriefing improved using a shortened guide. MethodsSenior medical students, nurse anesthesia students, and senior undergraduate nursing students participated in student operating room team training at a health sciences center in the southeastern United States. Student teams participated in a dual-scenario simulation-based training session with immediate after-action debriefings after each scenario. In 2018, student teams conducted the second debriefing using as a guide the teamwork assessment scale, an 11-item, 3-subscale, 6-point Likert-type instrument. In 2019, they used a shortened, revised, 5-item version of the teamwork assessment scale, the quick teamwork assessment scale. Trained observers rated the quality of the student-led debriefings using the Objective Structured Assessment of Debriefing, an 8-item, 5-point instrument. The Wilcoxon-Mann-Whitney test was used to compare the teamwork assessment scale-guided and the quick teamwork assessment scale-guided mean item debriefing scores. ResultsTwo observers rated 3 student-led team debriefings using the teamwork assessment scales as a guide in 2018, and 6 such debriefings happened using the quick teamwork assessment scale as a guide in 2019. For each debriefing, observer scores were averaged for each Objective Structured Assessment of Debriefing item; these mean scores were then averaged with other mean scores for each year. The use of the quick teamwork assessment scale resulted in a statistically significant higher mean score for the Analysis Objective Structured Assessment of Debriefing item compared with the use of the teamwork assessment scale (4.92 [standard deviation 0.20] versus 3.83 [standard deviation 0.76], P = .023). ConclusionThe use of a shortened teamwork assessment instrument as a debriefing guide for student teams in student operating room team training was more effective in analysis of actions than the original, longer tool. Next steps include determining the efficacy of the quick teamwork assessment scale in an actual clinical setting.

Brick in the wall? Linking quality of debriefing to participant learning in team training of interprofessional students

BackgroundThe evidence for the conventional wisdom that debriefing quality determines the effectiveness of learning in simulation-based training is lacking. We investigated whether the quality of debriefing in using simulation-based training...

Objective Structured Assessment of Debriefing (OSAD) in simulation-based medical education: Translation and validation of the German version.

Debriefing is essential for effective learning during simulation-based medical education. To assess the quality of debriefings, reliable and validated tools are necessary. One widely used validated tool is the Objective Structured Assessment of Debriefing (OSAD), which was originally developed in English. The aim of this study was to translate the OSAD into German, and to evaluate the reliability and validity of this German version (G-OSAD) according the 'Standards of Educational and Psychological Measurement'. In Phase 1, the validity evidence based on content was established by a multistage cross-cultural adaptation translation of the original English OSAD. Additionally, we collected expert input on the adequacy of the content of the G-OSAD to measure debriefing quality. In Phase 2, three trained raters assessed 57 video recorded debriefings to gather validity evidence based on internal structure. Interrater reliability, test-retest reliability, internal consistency, and composite reliability were examined. Finally, we assessed the internal structure by applying confirmatory factorial analysis. The expert input supported the adequacy of the content of the G-OSAD to measure debriefing quality. Interrater reliability (intraclass correlation coefficient) was excellent for the average ratings (three raters: ICC = 0.848; two raters: ICC = 0.790), and good for the single rater (ICC = 0.650). Test-retest reliability was excellent (ICC = 0.976), internal consistency was acceptable (Cronbach's α = 0.865), and composite reliability was excellent (ω = 0.93). Factor analyses supported the unidimensionality of the G-OSAD, which indicates that these G-OSAD ratings measure debriefing quality as intended. The G-OSAD shows good psychometric qualities to assess debriefing quality, which are comparable to the original OSAD. Thus, this G-OSAD is a tool that has the potential to optimise the quality of debriefings in German-speaking countries.

Sim for Life: Foundations-A Simulation Educator Training Course to Improve Debriefing Quality in a Low Resource Setting: A Pilot Study.

Despite the importance of debriefing, little is known about the effectiveness of training programs designed to teach debriefing skills. In this study, we evaluated the effectiveness of a faculty development program for new simulation educators at Mbarara University of Science and Technology in Uganda, Africa. Healthcare professionals were recruited to attend a 2-day simulation educator faculty development course (Sim for Life: Foundations), covering principles of scenario design, scenario execution, prebriefing, and debriefing. Debriefing strategies were contextualized to local culture and focused on debriefing structure, conversational strategies, and learner centeredness. A debriefing worksheet was used to support debriefing practice. Trained simulation educators taught simulation sessions for 12 months. Debriefings were videotaped before and after initial training and before and after 1-day refresher training at 12 months. The quality of debriefing was measured at each time point using the Objective Structured Assessment of Debriefing (OSAD) tool by trained, calibrated, and blinded raters. A total of 13 participants were recruited to the study. The mean (95% confidence interval) OSAD scores pretraining, posttraining, and at 12 months before and after refresher were 18.2 (14.3-22.1), 26.7 (22.8-30.6), 25.5 (21.2-29.9), and 27.0 (22.4-31.6), respectively. There was a significant improvement from pretraining to posttraining (P < 0.001), with no significant decay from posttraining to 12 months (P = 0.54). There was no significant difference in OSAD scores pre- versus post-refresher training at 12 months (P = 0.49). The Sim for Life Foundations program significantly improves debriefing skills with retention of debriefing skills at 12 months.

Development, Implementation, and Evaluation of a Faculty Development Workshop to Enhance Debriefing Skills Among Novice Facilitators.

IntroductionEffective debriefing during simulation-based training (SBT) is critical to promote learning outcomes. Despite debriefing’s central role in learning and various published debriefing methods and techniques, little is known about faculty development structure for debriefing training among novice facilitators. Continuing medical education courses often use simulation-based methods but provide minimal training in debriefing techniques to novice facilitators. We describe the development, implementation, and evaluation of a structured debriefing training workshop for novice facilitators.MethodsDesigned and conducted by simulation debriefing experts, a debriefing workshop was provided to novice facilitators serving as faculty during the simulation-based Sedation Provider Course (PC) at the 2018 Society of Pediatric Sedation conference. Emphasizing evidence-based key elements of effective debriefing, the workshop was divided into three components: 1) an introductory 30 minute didactic, 2) 75 minutes role modeling of simulated effective and ineffective debriefing 3) 120 minutes repetitive deliberate practice sessions with summative and formative feedback. Effective transfer of learned debriefing skills was assessed during facilitators’ PC debriefing using the Objective Structured Assessment of Debriefing (OSAD) tool, facilitators’ self-efficacy, and PC student learners’ evaluation of facilitator debriefings during the PC.ResultsSixteen facilitators participated in the 4-h workshop and the next day served as PC faculty. The median OSAD score was 31 (13-40) for all facilitators. OSAD components with lowest and highest performance were “Establishing Learning Environment” with a median score of 1 (1-5) and “Engagement of Learners,” with a median score of 4.75 (2.5-5). Facilitators' self- assessment in debriefing significantly improved on the 5-point Likert scale pre- and post-workshop, respectively. PC student learners’ evaluations revealed high degrees of satisfaction with debriefing quality.ConclusionsA proposed model integrating full-length debriefing and repetitive practice paired with summative and formative feedback provides a feasible and effective approach for debriefing training of novice facilitators for simulation-based educational courses.

Quality with quantity? Evaluating interprofessional faculty prebriefs and debriefs for simulation training using video

BackgroundHigh-fidelity simulation-based training is used increasingly for prelicensure student teams. Such sessions rely on faculty who are able to provide quality prebriefing and debriefing to foster learning among participants. We investigated how well faculty conducted prebriefing and debriefing as part of high-fidelity simulation-based training for interprofessional education. MethodsTwo trained observers independently rated 38 video-recorded sessions of combinations of 4 faculty conducting prebriefings and debriefings of prelicensure student teams after high-fidelity simulation-based training. Assessment was undertaken using the Objective Structured Assessment of Debriefing, an 8-item tool using a 5-point Likert scale (1 as minimum and 5 as maximum). Mean scores for each item were calculated. Inter-rater agreement was determined using Cohen’s kappa. A one-way between-subjects analysis of variance with post-hoc Tukey’s studentized range procedure was conducted to compare the effect of team facilitator grouping on the quality of team performance of each Objective Structured Assessment of Debriefing element during a prebriefing or a debriefing. Trend analyses of teams with 4 or more observations were performed using Kendall’s Tau coefficient test and linear regression analyses to identify whether teams showed improvement through time. Statistical significance was set at P < .05. ResultsA total of 7 combinations of faculty conducted between 1 to 14 prebriefings or debriefings. In general, faculty combinations performed better during debriefings compared with prebriefings, with only 1 team having 1 mean item score <3.50. Statistically significant differences between faculty combinations in mean item scores was more pronounced during the prebriefings (2 of 3 Objective Structured Assessment of Debriefing items rated) than during debriefings (1 of 8 Objective Structured Assessment of Debriefing items rated). Effect sizes were strong for all differences. Linear regression analysis revealed a statistically significant change through time for the 3 rated prebriefing items and for 7 of the 8 rated debriefing items. ConclusionInterprofessional faculty combinations in this study tended to have good quality prebriefings and debriefings. The quality of the prebriefings and debriefings can, however, be influenced by the composition of the facilitator teams, most prominently for prebriefings, and team performance does appear to change through time, especially during the debriefing. Future work will focus on whether the quality of prebriefings and debriefings influences learning by trainees.

EAssessment: development of an electronic version of the Objective Structured Assessment of Debriefing tool to streamline evaluation of video recorded debriefings.

The Objective Structured Assessment of Debriefing (OSAD) is an evidence-based, 8-item tool that uses a behaviorally anchored rating scale in paper-based form to evaluate the quality of debriefing in medical education. The objective of this project was twofold: 1) to create an easy-to-use electronic format of the OSAD (eOSAD) in order to streamline data entry; and 2) to pilot its use on videoed debriefings. The eOSAD was developed in collaboration with the LSU Health New Orleans Epidemiology Data Center using SurveyGizmo (Widgix Software, LLC, Boulder, CO, USA) software. The eOSAD was then piloted by 2 trained evaluators who rated 37 videos of faculty teams conducting pre-briefing and debriefing after a high-fidelity trauma simulation. Inter-rater reliability was assessed, and evaluators' qualitative feedback was obtained. Inter-rater reliability was good [prebrief, intraclass correlation coefficient, ICC = 0.955 (95% CI, 0.912-0.977), P < .001; debrief, ICC = 0.853 (95% CI, 0.713-0.924), P < .001]. Qualitative feedback from evaluators found that the eOSAD was easy to complete, simple to read and add comments, and reliably stored data that were readily retrievable, enabling the smooth dissemination of information collected. The eOSAD features a secure login, sharable internet access link for distant evaluators, and the immediate exporting of data into a secure database for future analysis. It provided convenience for end-users, produced reliable assessments among independent evaluators, and eliminated multiple sources of possible data corruption. The eOSAD tool format advances the post debriefing evaluation of videoed inter-professional team training in high-fidelity simulation.

Quality of interdisciplinary postsimulation debriefing: 360° evaluation

IntroductionDebriefing is widely perceived to be the most important component of simulation-based training. This study aimed to explore the value of 360° evaluation of debriefing by examining expert debriefing evaluators,...

Validation evidence of the paediatric Objective Structured Assessment of Debriefing (OSAD) tool

IntroductionDebriefing is essential to maximise the simulation-based learning experience, but until recently, there was little guidance on an effective paediatric debriefing. A debriefing assessment tool, Objective Structured Assessment of Debriefing...

Development of a tool to improve performance debriefing and learning: the paediatric Objective Structured Assessment of Debriefing (OSAD) tool

BackgroundSimulation is an important educational tool to improve medical training and patient safety. Debriefing after simulation is crucial to maximise learning and to translate the lessons learnt to improve real...

Operation Debrief

To explore the current status of performance feedback (debriefing) in the operating room and to develop and evaluate an evidence-based, user-informed intervention termed "SHARP" to improve debriefing in surgery. Effective debriefing is a key educational technique for optimizing learning in surgical settings. However, there is a lack of a debriefing culture within surgery. Few studies have prospectively evaluated educational interventions to improve the quality and quantity of performance feedback in surgery. This was a prospective pre- and post-study of 100 cases involving 22 trainers (attendings) and 30 surgical residents (postgraduate years 3-8). A trained researcher assessed the quality of debriefings provided to the trainee using the validated Objective Structured Assessment of Debriefing (OSAD) tool alongside ethnographic observation. Following the first 50 cases, an educational intervention termed "SHARP" was introduced and measures repeated for a further 50 cases. User satisfaction with SHARP was assessed via questionnaire. Twenty percent of the cases were observed independently by a second researcher to test interrater reliability. Interrater reliability for OSAD was excellent (ICC = 0.994). Objective scores of debriefing (OSAD) improved significantly after the SHARP intervention: median pre = 19 (range, 8-31); median post = 33 (range, 26-40), P < 0.001. Strong correlations between observer (OSAD) and trainee rating of debriefing were obtained (median ρ = 0.566, P < 0.01). Ethnographic observations also supported a significant improvement in both quality and style of debriefings. Users reported high levels of satisfaction in terms of usefulness, feasibility, and comprehensiveness of the SHARP tool. SHARP is an effective and efficient means of improving performance feedback in the operating room. Its routine use should be promoted to optimize workplace-based learning and foster a positive culture of debriefing and performance improvement within surgery.

Objective Structured Assessment of Debriefing

: To identify the features of effective debriefing and to use this to develop and validate a tool for assessing such debriefings. : Simulation-based training has become an accepted means of surgical skill acquisition. A key component of this is debriefing-yet there is a paucity of research to guide best practice. : Phase 1-Identification of best practice and tool development. A search of the Medline, Embase, PsycINFO, and ERIC databases identified current evidence on debriefing. End-user input was obtained through 33 semistructured interviews conducted with surgeons (n = 18) and other operating room personnel (n = 15) from 3 continents (UK, USA, Australia) using standardized qualitative methodology. An expert panel (n = 7) combined the data to create the Objective Structured Assessment of Debriefing (OSAD) tool. Phase 2-Psychometric testing. OSAD was tested for feasibility, reliability, and validity by 2 independent assessors who rated 20 debriefings following high-fidelity simulations. : Phase 1: 28 reports on debriefing were retrieved from the literature. Key components of an effective debriefing identified from these reports and the 33 interviews included: approach to debriefing, learning environment, learner engagement, reaction, reflection, analysis, diagnosis of strengths and areas for improvement, and application to clinical practice. Phase 2: OSAD was feasible, reliable [inter-rater ICC (intraclass correlation coefficient) = 0.88, test-retest ICC = 0.90], and face and content valid (content validity index = 0.94). : OSAD provides an evidence-based, end-user informed approach to debriefing in surgery. By quantifying the quality of a debriefing, OSAD has the potential to identify areas for improving practice and to optimize learning during simulation-based training.