Behavioural Marker System Research Articles

Training and assessment of non-technical skills in Norwegian helicopter emergency services: a cross-sectional and longitudinal study

BackgroundDeficient non-technical skills (NTS) among providers of critical care in helicopter emergency medical services (HEMS) is a threat to patient and operational safety. Skills can be improved through simulation-based training and assessment. A previous study indicated that physicians underwent less frequent training compared to pilots and HEMS crew members (HCM) and that all professional groups in Norwegian HEMS received limited training in how to cope with fatigue. Since then, training initiatives and a fatigue risk management project has been initiated. Our study aimed to explore if the frequency of simulation-based training and assessment of NTS in Norwegian HEMS has changed since 2011 following these measures.MethodsA cross-sectional web-based survey from October through December 2016, of physicians, HCM and pilots from all civilian Norwegian HEMS-bases reporting the overall extent of simulation-based training and assessment of NTS.ResultsOf 214 invited, 109 responses were eligible for analysis. The frequency of simulation-based training and assessment of NTS has increased significantly for all professional groups in Norwegian HEMS, most prominently for the physicians. For all groups, the frequency of assessment is generally lower than the frequency of training.ConclusionsPhysicians in Norwegian HEMS seem to have adjusted to the NTS training culture of the other crew member groups. This might be a consequence of improved NTS training programs. The use of behavioural marker systems systematically in HEMS should be emphasized.

Validation of a Behavioral Marker System for Rating Cadet’s Non-Technical Skills

Simulator-based training assumes a very important role in the maritime domain, particularly in the education of Officers Of the Watch (OOW). In the Portuguese Navy, most of the cadet's skills as future OOW rely on the success of this training. Beyond theory and technical training, the development of nontechnical skills is a key factor for obtaining officers capable of identifying and solving problems. To optimize the training and development of non-technical skills, using the Portuguese Naval Academy Simulator, a previously designed Behavioral Marker System model was further assessed. The revised model, which comprises new parameters such as the effectiveness of the task, was validated through a set of simulated sessions, where 11 instructors and 48 students participated. After each session, data was collected with questionnaires and focus group discussion, focusing on the quality and usability of the model and on the design of the scenario. The results show that the revised model, positively addresses the limitations found on the previous version, and it has received encouraging feedback from both instructors and cadets. This new model is now under implementation in all the Naval Academy course programs, and future research aims to digitalize the behavior markers.

Evaluation of a Measurement System to Assess ICU Team Performance.

Measuring teamwork is essential in critical care, but limited observational measurement systems exist for this environment. The objective of this study was to evaluate the reliability and validity of a behavioral marker system for measuring teamwork in ICUs. Instances of teamwork were observed by two raters for three tasks: multidisciplinary rounds, nurse-to-nurse handoffs, and retrospective videos of medical students and instructors performing simulated codes. Intraclass correlation coefficients were calculated to assess interrater reliability. Generalizability theory was applied to estimate systematic sources of variance for the three observed team tasks that were associated with instances of teamwork, rater effects, competency effects, and task effects. A 15-bed surgical ICU at a large academic hospital. One hundred thirty-eight instances of teamwork were observed. Specifically, we observed 88 multidisciplinary rounds, 25 nurse-to-nurse handoffs, and 25 simulated code exercises. No intervention was conducted for this study. Rater reliability for each overall task ranged from good to excellent correlation (intraclass correlation coefficient, 0.64-0.81), although there were seven cases where reliability was fair and one case where it was poor for specific competencies. Findings from generalizability studies provided evidence that the marker system dependably distinguished among teamwork competencies, providing evidence of construct validity. Teamwork in critical care is complex, thereby complicating the judgment of behaviors. The marker system exhibited great potential for differentiating competencies, but findings also revealed that more context specific guidance may be needed to improve rater reliability.

The Development of a Skills Taxonomy for Nursing Crisis Management

BackgroundEffective crisis management requires nurses to demonstrate proper nontechnical skills as first responders to intervene early in handling crisis situations. Currently, a behavioral marker system with structured taxonomy of nontechnical skills in crisis management is lacking in the nursing literature. This article outlines the development of a skills taxonomy, which will form the basis of the behavioral marker system that we aim to develop in our next phase of study. MethodsWe audio-recorded 50 registered nurses and enrolled nurses over five simulation debriefing sessions. We analyzed the data using a thematic analysis approach and developed a skills taxonomy. To ensure face validation, we invited four experts to review the skills taxonomy and to comment on each behavior's observability. ResultsWe developed a skills taxonomy comprising six categories and 18 underlying observable behaviors (elements). The developed categories are task and resource management, situational awareness, teamwork, communication, control of emotions, and leadership. ConclusionWe have developed a skills taxonomy fostering the use of a common terminology for nontechnical skills in nursing crisis-management teams. Further work to develop a rating system and to test its reliability will be conducted.

Medical Students’ Non-Technical Skills (Medi-StuNTS): preliminary work developing a behavioural marker system for the non-technical skills of medical students in acute care

IntroductionGood non-technical skills (NTS) are critical to the delivery of high-quality patient care. It is increasingly recognised that training in such skills should be incorporated into primary medical training curricula. This study aimed to develop an NTS behavioural marker system (BMS), specifically applicable to medical students, for use within simulated acute care scenarios.MethodsThe methodology used to develop other BMS was adopted and modified. Following ethical approval, 16 final year medical students participated in acute care simulated scenarios. Semistructured interviews were performed to gauge the understanding of NTS. A panel meeting of subject matter experts was convened to translate key NTS into skill elements and observable behaviours. A second expert panel was consulted to refine aspects of the BMS. Further refinement and initial face validity was undertaken by a third panel of experts using the prototype BMS to observe prerecorded simulation scenarios.ResultsFive categories of NTS were identified: situation awareness, teamwork and communication, decision-making and prioritisation, self-awareness, and escalating care. Observable behaviours in each category describe good and poor performance. Escalating care was identified as a unique component that incorporated behaviours related to each of the other four skill categories. A 5-point rating scale was developed to enable both peer-to-peer and tutor-to–student feedback.ConclusionThe Medi-StuNTS (Medical Students’ Non-Technical Skills) system is the first BMS for the NTS of medical students. It reinforces the importance of escalating care effectively. It provides an exciting opportunity to provide feedback to medical students and may ultimately aid their preparedness for professional practice.

Validity evidence of non-technical skills assessment instruments in simulated anaesthesia crisis management.

We sought to evaluate the validity of two non-technical skills evaluation instruments, the Anaesthetists' Non-Technical Skills (ANTS) behavioural marker system and the Ottawa Global Rating Scale (GRS), to apply them to anaesthesia training. The content validity, response process, internal structure, relations with other variables and consequences were described for validity evidence. Simulated crisis management sessions were initiated during which two trained raters evaluated the performance of postgraduate first-, second- and third-year (PGY-1, PGY-2 and PGY-3) anaesthesia residents. The study included 70 participants, composed of 24 PGY-1, 24 PGY-2 and 22 PGY-3 residents. Both instruments differentiated the non-technical skills of PGY-1 from PGY-3 residents (P <0.05). Inter-rater agreement was measured using the intraclass correlation coefficient (ICC). For the ANTS instrument, the intraclass correlation coefficients for task management, team-working, situation awareness and decision-making were 0.79, 0.34, 0.81 and 0.70, respectively. For the Ottawa GRS, the intraclass correlation coefficients for overall performance, leadership, problem-solving, situation awareness, resource utilisation and communication skills were 0.86, 0.83, 0.84, 0.87, 0.80 and 0.86, respectively. The Cronbach's alpha for internal consistency of the ANTS instrument was 0.93, and was 0.96 for the Ottawa GRS. There was a high correlation between the ANTS and Ottawa GRS. The raters reported the ease of use of the Ottawa GRS compared to the ANTS. We found sufficient evidence of validity in the ANTS instrument and the Ottawa GRS for the evaluation of non-technical skills in a simulated anaesthesia setting, but the Ottawa GRS was more practical and had higher reliability.

Development of a Behavioural Marker System for Rating Cadet’s Non-Technical Skills

Despite the adoption of crew resource management training for bridge teams over the last decades, the training is still heavily focussed on technical achievements. In an educational context, the situation is more problematical, since with requirement of developing the technical skills, there is a need to build and evaluate the non-technical skills of cadets with little experience in bridge team management. In parallel with the application of team leadership models, the Portuguese Naval Academy conducted a research to improve the development and assessment of non-technical skills in bridge simulators. This paper describes the method used to identify the key non-technical skills required for naval cadets and to develop a behavioural marker system for their measurement. A literature review of behavioural marker systems was supplemented with an analysis of interviews conducted with students and simulator instructors. Additionally, further analysis of Portuguese Navy accidents reports was made, applying the HFACS framework to identify the relevant non-technical skills involved in the accidents. The resulting rating system covers five skill categories (leadership, situational awareness, communication, team work and decision making), each one with three rating elements. The framework is currently under evaluation tests in bridge simulators sessions, within an educational context.

Aiming for excellence - A simulation-based study on adapting and testing an instrument for developing non-technical skills in Norwegian student nurse anaesthetists.

There is increasing focus on building safety into anaesthesia practice, with excellence in anaesthesia as an aspirational goal. Non-technical skills are an important factor in excellence and improved patient safety, though there have been few systematic attempts at integrating them into anaesthesia nursing education. This study aimed to test the reliability of NANTS-no, a specially adapted behavioural marker system for nurse anaesthetists in Norway, and explore the development of non-technical skills in student nurse anaesthetists. The pre-test post-test design incorporated a 10-week simulation-based programme, where non-technical skills in 14 student nurse anaesthetists were rated on three different occasions during high-fidelity simulation, before and after taking part in a training course. NANTS-no demonstrated high overall inter-rater reliability (ICC=0.91), high test-retest reliability (ICC=0.94) and good internal consistency (Cronbach's α of 0.85-0.92). A significant improvement was demonstrated across all categories of non-technical skills, with greatest improvements between the first and third and second and third sessions. There was also a significant improvement in two categories between the first and second sessions. NANTS-no is therefore suitable for assessing non-technical skills during simulation training in anaesthesia nursing education. More research is needed to validate its use in clinical practice.

The importance of non‐technical skills and risk reduction in the operating theatre

Key content Preventable harm still occurs at unacceptable levels in the operating theatre and this is often due to systems failure. Systems failures are often the result of non‐technical errors. Identifying and capturing errors in the system is important for improving patient safety. Initiatives such as checklists should be supported. Distractions and interruptions in the operating theatre should be limited. Non‐technical skills and teamwork training are important in this area. Behavioural marker systems may be useful in evaluating non– technical skills. Learning objectives To identify risk areas in the operating theatre environment. To describe ways to identify and capture error. To describe the role of training and teamwork in reducing risk. Ethical issues How do we identify risks and raise concerns?

Development of a Behavioral Marker System to Assess Intensive Care Unit Team Performance

Teamwork plays a pivotal role in ensuring the quality and safety of care delivery in the intensive care unit (ICU). The measurement of these skills, therefore, is paramount to quantifying performance, providing developmental feedback, and evaluating the impact of initiatives to improve performance. The purpose of this paper is to detail the development of a theoretically-driven and context-driven approach to the measurement of teamwork skills in critical care.

The human factor in agriculture: An interview study to identify farmers’ non-technical skills

BackgroundFarming represents a high risk occupation, responsible for several thousand worker injuries and fatalities worldwide per year. Research within other high risk industries, such as aviation, shipping and healthcare, has identified the importance of non-technical skills in maintaining effective performance and reducing the risk of an adverse event. The aim of the current study was to identify the categories of non-technical skills that are typically used by farmers. MethodA sample of 32 farmers, from within two geographical regions (Scotland and Northern Ireland), were interviewed using the critical incident technique. The interview transcripts were then coded using thematic analysis in order to identify reported non-technical skills. ResultsParticipants reported the daily use of a range of non-technical skills, these differed according to whether the farmer was working as part of a team or alone. Team non-technical skills were identified as: situation awareness, decision-making, leadership, teamwork and task management. Lone worker non-technical skills were identified as: situation awareness, decision-making and task management. ConclusionThe results indicate that non-technical skills are an important aspect of farmers’ work performance and safety; mirroring the findings reported within other high risk industries. Further research is required to validate the skill set suggested here, and to develop a behavioural marker system similar to that used in other industries.

Is it time for a nontechnical skills approach to prescribing?

Is it time for a nontechnical skills approach to prescribing?

A systematic review of behavioural marker systems in healthcare: what do we know about their attributes, validity and application?

ObjectiveBehavioural marker systems are advocated as a method for providing accurate assessments, directing feedback and determining the impact of teamwork improvement initiatives. The present article reports on the state of...

A description of communication patterns during CPR in ICU

BackgroundDeficiencies in communication in health care are a common source of medical error. Preferred communication patterns are a component of resuscitation teaching. We audio-recorded resuscitations in a mixed paediatric medical and surgical ICU to describe communication. MethodsIn the intensive care unit, resuscitation events were prospectively audio-recorded by two trained observers (using handheld recorders). Recordings were transcribed and anonymised within 24h. We grouped utterances regarding the same subject matter from beginning (irrespective of response) as a communication epoch. For each epoch, we describe the initiator, audience and content of message. Teamwork behaviours were described using Anesthesia Nontechnical Skills framework (ANTS), a behavioural marker system for crisis-resource management. ResultsConsent rates from staff were 139/140 (99%) and parents were 67/92 (73%). We analysed 36min 57s of audio dialogue from 4 cardiac arrest events in 363h of prospective screening. There were 180 communication epochs (1 every 12s): 100 (56%) from the team-leader and 80 (44%) from non-team-leader(s). Team-leader epochs were to give or confirm orders or assert authority (61%), clarify patient history (14%) and provide clinical updates (25%). Non-team-leader epochs were more often directed to the team (65%) than the team-leader (35%). Audio-recordings provided information for 80% of the ANTS component elements with scores of 2–4. ConclusionCommunication epochs were frequent, most from the team-leader. We identified an ‘outer loop’ of communication between team members not including the team-leader, responsible for 44% of all communication events. We discuss difficulties in this research methodology. Future work includes exploring the process of the ‘outer loop’ by resuscitation team members to evaluate the optimal balance between single leader and team suggestions, the content of the outer loop discussions and in-event communication strategies to improve outcomes.

Assessing neurosurgical non-technical skills: an exploratory study of a new behavioural marker system.

The management of non-technical skills is a major factor affecting teamwork quality and patient safety. This article presents a behavioural marker system for assessing neurosurgical non-technical skills (BMS-NNTS). We tested the BMS during deep brain stimulation surgery. We developed the BMS in three stages. First, we drew up a provisional assessment tool based on the literature and observation tools developed for other surgical specialties. We then analysed videos made in an operating room (OR) during deep brain stimulation operations in order to ensure there were no significant omissions from the skills list. Finally, we used five videos of operations to identify the behavioural markers of non-technical skills in verbal communications. Analyses of more than six hours of observations revealed 3515 behaviours from which we determined the neurosurgeon's non-technical skills behaviour pattern. The neurosurgeon frequently engaged in explicit coordination, situation awareness and leadership behaviours. In addition, the neurosurgeon's behaviours differed according to the stage of the operation and the OR staff members with whom she was communicating. Our behavioural marker system provides a structured approach to assessing non-technical skills in the field of neurosurgery. It can also be transferred to other surgical specialties and used in surgeon training curricula.

Assessing Nurse Anaesthetists' Non‐Technical Skills in the operating room

Incident reporting and fieldwork in operating rooms have shown that some of the errors that arise in anaesthesia relate to inadequate use of non-technical skills. To provide a tool for training and feedback on nurse anaesthetists' non-technical skills, this study aimed to adapt the Anaesthetists' Non-Technical Skills (ANTS) as a behavioural marker system for the formative assessment of nurse anaesthetists' non-technical skills in the operating room. A qualitative approach with focus group interviews was used to identify the non-technical skills of nurse anaesthetists in the operating room. The interview data were transcribed verbatim. Directed content analysis was used to code and sort data deductively into the ANTS categories: task management, team working, situation awareness and decision making. The prototype named Nurse Anaesthetists' Non-Technical Skills (N-ANTS) was presented and discussed in a group of subject matter experts to ensure face validity. The N-ANTS system consists of the same four categories as ANTS and 15 underlying elements. Three to five good and poor behavioural markers for each element were identified. The headings and definitions of the categories and elements were adjusted to encompass the behavioural markers in N-ANTS. The differences that emerged mainly reflected statements regarding the establishment of role, competence, and task delegation. A behavioural marker system, N-ANTS, for nurse anaesthetists was adapted from a behavioural marker system, ANTS, for anaesthesiologists.

Strategies in coping with complexity: Development of a behavioural marker system for air traffic controllers

To meet the increasing air traffic volume, organizations seek better tools to assess the traffic handling capacity of air traffic control (ATC) systems. This effort requires a better understanding of how complex situations are related to controller strategies and how controllers intervene to maintain control. Recent empirical studies and reviews have shown that controllers cope with complexity by adapting priorities, managing their cognitive resources, and regulating their own performance. This study discusses the development of a behavioural marker system to evaluate and provide feedback on the strategies that controllers use to cope with complexity. An extensive literature review provided the basis for an initial classification of strategies for coping with complexity which was tested in an observational study for reliability. After three iterations of adaptation to the taxonomy, two independent raters were able to reach acceptable levels of reliability in classifying video segments of simulated traffic scenarios. A potential application of the study regards the design of refresher training enriched with the principles of error management and the assessment of new technologies and controller tools on the handling capacity of ATC systems.

Development of a behavioural marker system for scrub practitioners' non‐technical skills (SPLINTS system)

Adverse events still occur despite ongoing efforts to reduce harm to patients. Contributory factors to adverse events are often due to limitations in clinicians' non-technical skills (e.g. communication, situation awareness), rather than deficiencies in technical competence. We developed a behavioural rating system to provide a structured means for teaching and assessing scrub practitioners' (i.e. nurse, technician, operating department practitioner) non-technical skills. Psychologists facilitated focus groups (n = 4) with experienced scrub practitioners (n = 16; 4 in each group) to develop a preliminary taxonomy. Focus groups reviewed lists of non-technical-skill-related behaviours that were extracted from an interview study. The focus groups labelled skill categories and elements and also provided examples of good and poor behaviours for those skills. An expert panel (n = 2 psychologists; n = 1 expert nurse) then used an iterative process to individually and collaboratively review and refine those data to produce a prototype skills taxonomy. A preliminary taxonomy containing eight non-technical skill categories with 28 underlying elements was produced. The expert panel reduced this to three categories (situation awareness, communication and teamwork, task management), each with three underlying elements. The system was called the Scrub Practitioners' List of Intraoperative Non-Technical Skills system. A scoring system and a user handbook were also developed. A prototype behavioural rating system for scrub practitioners' non-technical skills was developed, to aid in teaching and providing formative assessment. This important aspect of performance is not currently explicitly addressed in any educational route to qualify as a scrub practitioner.

Development and Validation of a Tool to Assess Emergency Physicians' Nontechnical Skills

Nontechnical skills are "the cognitive, social and personal resource skills that complement technical skills, and contribute to safe and efficient task performance." Our research team developed and evaluated the task of developing and validating a behavioral marker system for the observational assessment of emergency physicians' nontechnical skills. The development of the tool was divided into 3 phases and used triangulation of data from a number of sources. During phase 1, a provisional assessment tool was developed according to published literature and curricula. Phase 2 used analysis of staff interviews and field observations to determine whether the skill list contained any significant omissions. These studies were also used to identify behavioral markers linked to nontechnical skills in the context of the emergency department (ED) and establish whether skills included in the tool were observable. Phase 3 involved evaluating the content validity index of exemplar behaviors, using a survey of experts. A behavioral marker system was developed that comprised 12 emergency medicine-specific nontechnical skills, grouped into 4 categories. Content validity was assessed with a survey of 148 emergency medicine staff, and 75% of items achieved the recommended content validity index greater than 0.75. Data from the survey enabled further refinement of the behavioral markers to produce a final version of the tool. Although further evaluative studies are needed, this behavioral marker system provides a structured approach to the assessment and training of nontechnical skills in the ED.

The development of a prototype behavioral marker system for US Navy officers of the deck

The officer of the deck (OOD) of a US Navy ship is in charge of the safe and proper operation of the ship, and accountable to the commanding officer for every event that occurs during his or her OOD watch. This paper discusses the development of a prototype behavioral marker system to evaluate, and provide feedback on, the nontechnical (cognitive, social, and personal resource) skills of OODs. An initial set of 17 categories of nontechnical skills were identified from a literature review. A focus group with four qualified OODs used the skills identified from the literature review to develop an initial taxonomy of five categories, each with two or three corresponding behavioral elements. This taxonomy was then used to classify 149 statements concerned with the nontechnical skills of OODs collected from 16 critical incident interviews. After three iterations of adaptations to the taxonomy, two independent raters were able to reach acceptable levels of reliability in using the taxonomy to classify the statements. Although further development work is required, it is suggested that the prototype behavioral marker system has implications for improving safety and performance on military and civilian ships.