Non-technical Skills Performance Research Articles

Evaluation of the Scrub Practitioners’ List of Intraoperative Non-Technical Skills (SPLINTS) system

BackgroundThe Scrub Practitioners’ List of Intraoperative Non-Technical Skills (SPLINTS) system is a new tool for training and assessing scrub practitioner (nurse, technician) behaviours during surgical operations. ObjectivesThe aim of the study was to test the psychometric properties including inter-rater reliability of the prototype SPLINTS behavioural rating system. MethodsExperienced scrub practitioners (n=34) attended a one-day session where they received background training in human factors and non-technical skills and were also trained to use the SPLINTS system. They then used SPLINTS to rate the scrub practitioners’ non-technical skill performance in seven standardized simulated, surgical scenarios. ResultsReliability, measured by within-group agreement (rwg) for the three skill categories and six out of nine elements, was acceptable (rwg>0.7). Participants were within one scale point of expert ratings in >90% of skill categories and elements, and could use SPLINTS to score performance with a reasonable level of accuracy. There was good internal consistency of the system: absolute mean difference was M<0.2 of a scale point for all three categories. Participants were surveyed and they indicated that the system was complete and usable as an assessment tool. ConclusionThe reliability of the SPLINTS system was deemed to be adequate for assessing scrub practitioners’ non-technical skills in simulated, standardized, video scenarios. On the basis of these results, the system can now move on to usability testing in the real operating theatre.

Evaluation of the SPLINTS system for scrub practitioners' non-technical skills

Background: The Scrub Practitioners’ List of Intraoperative NonTechnical Skills (SPLINTS) system is a new tool for training and assessing scrub practitioner behaviours during surgical operations. Method: Experienced scrub practitioners (n = 34) attended a single day session where they received background training in human factors and nontechnical skills and were trained to use the SPLINTS system. They then rated the scrub practitioners’ nontechnical skill performance in seven videorecorded simulated scenarios and completed a posttraining questionnaire. Results: Withingroup agreement for all three skill categories, and for six of the nine elements, was acceptable (rwg>.70). Participants could use SPLINTS to accurately score performance compared with expert ratings and were within one scale point of expert ratings in > 90% of skill categories and elements. There was good internal consistency between elements and their categories (M<.2 of a scale point) and participants reported that the system was complete. Conclusion: The SPLINTS system was deemed adequately reliable, for rating scrub practitioners’ nontechnical skills, using standardized filmed scenarios. Usability of the SPLINTS system in the real operating theatre environment is still to be assessed.

Non-technical skills assessment in the post-operative setting

Introduction Communication failures, poor decision-making or a lack of leadership may result in negative patient outcomes in the operating room but also on the ward/emergency room setting. Assessment tools such as the Non Technical Skills for Surgeons (NOTSS), the revised Non technical Skills (NOTECHS) Scale or the Ottawa Global Rating Scale (GRS), have only been applied to operating room/critical care settings. We assessed the construct validity and inter-rater reliability of these tools in a simulated surgical ward environment. Methods Twenty surgical residents (PGY1-2) were video-recorded managing a simulated post-operative complication using a high-fidelity full body simulator in a mock surgical ward. Subjects were randomized to one of 3 scenarios (hemorrhagic shock/septic shock/pulmonary embolism). Two independent experts rated subjects' non-technical skills performance using the NOTSS, NOTECHS and Ottawa GRS. For each scale, the intra-class correlation (ICC), (two-way mixed effect, single measures) measured the inter-rater reliability and Mann-Whitney U test was compared scores across resident training level. Results Inter-rater reliability was strong for the NOTSS (ICC=0.75, p Conclusions We demonstrated the construct validity and inter-rater reliability of existing non-technical rating scales in a post-operative crisis management domain. Future surgical curricula might incorporate these scales to assess residents' non-technical skills during post-operative crisis management.

Looking in the mirror: Self-debriefing versus instructor debriefing for simulated crises*

To examine the effectiveness of self-debriefing as compared to instructor debriefing in the change of nontechnical skills performance of anesthesiology residents. Prospective, randomized, controlled study. A university hospital simulation center. : Fifty anesthesiology residents. Subjects were instructed in the principles of nontechnical skills for crisis management. Subsequently, each resident participated in a high-fidelity simulated anesthesia crisis scenario (pretest). Participants were randomized to either a video-assisted self-debriefing or instructor debriefing. In the self-debriefing group, subjects reviewed their pretest scenario by themselves, guided by the Anesthetists' Non-Technical Skills scale. The instructor debriefing group reviewed their pretest scenario guided by an expert instructor also using the Anesthetists' Non-Technical Skills scale as a framework. Immediately following their respective debriefings, subjects managed a second simulated crisis (post-test). After all data were collected, two blinded experts independently rated videos of all performances in a random order using the Anesthetists' Non-Technical Skills scale. Performance significantly improved from pretest to post-test (p < .01) regardless of the type of debriefing received. There was no significant difference in the degree of improvement between self-debriefing and instructor debriefing (p = .58). Nontechnical skills for crisis resource management improved with training, as measured by the Anesthetists' Non-Technical Skills scale. Crisis resource management can be taught, with measurable improvements. Effective teaching of nontechnical skills can be achieved through formative self-assessment even when instructors are not available.

Intermethod Reliability of Real‐time Versus Delayed Videotaped Evaluation of a High‐fidelity Medical Simulation Septic Shock Scenario

High-fidelity medical simulation (HFMS) is increasingly utilized in resident education and evaluation. No criterion standard of assessing performance currently exists. This study compared the intermethod reliability of real-time versus videotaped evaluation of HFMS participant performance. Twenty-five emergency medicine residents and one transitional resident participated in a septic shock HFMS scenario. Four evaluators assessed the performance of participants on technical (26-item yes/no completion) and nontechnical (seven item, five-point Likert scale assessment) scorecards. Two evaluators provided assessment in real time, and two provided delayed videotape review. After 13 scenarios, evaluators crossed over and completed the scenarios in the opposite method. Real-time evaluations were completed immediately at the end of the simulation; videotape reviewers were allowed to review the scenarios with no time limit. Agreement between raters was tested using the intraclass correlation coefficient (ICC), with Cronbach's alpha used to measure consistency among items on the scores on the checklists. Bland-Altman plot analysis of both conditions revealed substantial agreement between the real-time and videotaped review scores by reviewers. The mean difference between the reviewers was 0.0 (95% confidence interval [CI] = -3.7 to 3.6) on the technical evaluation and -1.6 (95% CI = -11.4 to 8.2) on the nontechnical scorecard assessment. Comparison of evaluations for the videotape technical scorecard demonstrated a Cronbach's alpha of 0.914, with an ICC of 0.842 (95% CI = 0.679 to 0.926), and the real-time technical scorecard demonstrated a Cronbach's alpha of 0.899, with an ICC of 0.817 (95% CI = 0.633 to 0.914), demonstrating excellent intermethod reliability. Comparison of evaluations for the videotape nontechnical scorecard demonstrated a Cronbach's alpha of 0.888, with an ICC of 0.798 (95% CI = 0.600 to 0.904), and the real-time nontechnical scorecard demonstrated a Cronbach's alpha of 0.833, with an ICC of 0.714 (95% CI = 0.457 to 0.861), demonstrating substantial interrater reliability. The raters were consistent in agreement on performance within each level of training, as the analysis of variance demonstrated no significant differences between the technical scorecard (p = 0.176) and nontechnical scorecard (p = 0.367). Real-time and videotaped-based evaluations of resident performance of both technical and nontechnical skills during an HFMS septic shock scenario provided equally reliable methods of assessment.

Inefficacy of simulator‐based training on anaesthesiologists' non‐technical skills*

Analysis of critical incidents in anaesthesia practice emphasizes the important role of non-technical skills (NTS) in improving patient safety. Therefore, debriefing with regard to NTS has been added to medical management (MM) simulator training to improve acute crisis competencies of anaesthesiologists. The purpose of this study was to compare the NTS and MM performance of two groups of anaesthesiologists in a simulated anaesthesia crisis after undergoing different types of training. Forty-two anaesthesiologists were randomly assigned to two different training groups, with each group participating in two simulation training sessions. One group's (NTS+MMG) training included extensive debriefing of NTS (resource management, planning, leadership and communication) and MM, while the other group (MMG) received a simpler debriefing that focused solely on MM. The quality and quantity of NTS and quality of MM performance were rated by reviewing the videotapes of the scenarios. NTS+MMG did not prove superior to MMG with regard to behavioural and MM markers. The quality of NTS performances correlated significantly (P<0.01) with the quantity of NTS demonstration, and also correlated significantly with the MM actions (P<0.01) in both groups. A single session of training including debriefing of NTS and MM did not improve the NTS performances of anaesthesiologists when compared with anaesthesiologists who only received MM training. This might indicate that a more frequent or individual training is needed to improve participants' NTS performance.

Is team training in briefings for surgical teams feasible in simulation?

We developed a module for surgical team training using briefings in simulated crisis scenarios and here we report preliminary findings. Nine surgical teams (34 trainees) participated in a pre-training simulation, followed by an interactive workshop on briefing and checklists, and then a post-training simulation. Both technical and non-technical skills were assessed via observation during simulations by expert trainers who provided feedback on performances at the end of simulation. Trainees also reported their attitudes to briefings and evaluated the training. Pre-training attitudes to briefing were positive, some of which improved post-training and trainees’ evaluation of the training was positive. Surgeons’ technical skill improved significantly post-training, but their decision-making skill was rated lower than other non-technical skills, compared to other trainees. The training did not appear to greatly improve non-technical skill performance. Training surgical teams in simulation is feasible but much more work is needed on measurement development and training strategy to confirm its efficacy and utility.

Value of Debriefing during Simulated Crisis Management

The debriefing process during simulation-based education has been poorly studied despite its educational importance. Videotape feedback is an adjunct that may enhance the impact of the debriefing and in turn maximize learning. The purpose of this study was to investigate the value of the debriefing process during simulation and to compare the educational efficacy of two types of feedback, oral feedback and videotape-assisted oral feedback, against control (no debriefing). Forty-two anesthesia residents were enrolled in the study. After completing a pretest scenario, participants were randomly assigned to receive no debriefing, oral feedback, or videotape-assisted oral feedback. The debriefing focused on nontechnical skills performance guided by crisis resource management principles. Participants were then required to manage a posttest scenario. The videotapes of all performances were later reviewed by two blinded independent assessors who rated participants' nontechnical skills using a validated scoring system. Participants' nontechnical skills did not improve in the control group, whereas the provision of oral feedback, either assisted or not assisted with videotape review, resulted in significant improvement (P < 0.005). There was no difference in improvement between oral and video-assisted oral feedback groups. Exposure to a simulated crisis without constructive debriefing by instructors offers little benefit to trainees. The addition of video review did not offer any advantage over oral feedback alone. Valuable simulation training can therefore be achieved even when video technology is not available.

Individual Debriefing vs. Computer Based Multimedia Instruction After Patient Crisis Simulation

BACKGROUND: Several advantages of simulation over didactic and patient dependent teaching have lead to a proliferation of patient simulators in academic centers. Although simulation experience alone seems insufficient for learning, research evaluating different modes of instruction following simulation is lacking (1). Standardized multimedia has shown to be useful in teaching surgical skills, but has not been evaluated for use as an adjunct in crises management training. The primary purpose of this study is to determine whether standardized computer-based multimedia instruction is effective for learning in patient crises scenarios. The secondary purpose is to compare multimedia debriefing to personal oral debriefing with an expert. METHODS: Thirty anesthesia residents were recruited to manage three different simulated resuscitation crises using a high-fidelity patient simulator. Following the first scenario, subjects were randomized to either a computer-based multimedia tutorial or a personal debriefing with an expert followed by a posttest and retention test five weeks later. To date, the performances of 20 residents were independently rated by two expert blinded assessors using the previously validated Anesthesia Non Technical Skills (ANTS) marking system (2). RESULTS: Resident performance of non-technical skills improved significantly in both groups compared to pretest p<0.05). The improvement was sustained after five weeks for both groups (p<0.05)(see Figure). No significant difference in the performance of residents receiving either type of debriefing was demonstrated in any of the tests.FigureCONCLUSION: Computer based multimedia instruction is an effective method of teaching non-technical skills in simulated crisis scenarios and may be as effective as oral personalized debriefing. Multimedia may be a valuable adjunct to centers when debriefing expertise is not available. Multimedia may also improve simulation utilization by reducing anxiety of peer evaluation in continuing medical education. (1) Savodelli GL et al., submitted 2005. (2) Fletcher G et al., Br J Anaesth 2003; 90 (5) 580–588. Conflict of Interest: Authors indicated they have nothing to disclose.