Training and assessment of non-technical skills in the operating theatre: where next?

Abstract

The importance of human factors and non-technical skills in the management of emergencies was initially recognised by the aviation industry as a result of retrospective analysis of catastrophic accidents 1. The concept that major errors resulted from poor non-technical skills rather than the lack of technical ability by members of the air crew helped to promote the role of training and assessment of non-technical skills for aviation staff as a key measure in preventing errors in this industry. There is good evidence that crew (later, crisis) resource management (CRM) training and assessment for non-technical skills improves attitudes to error avoidance in aviation 2, although there is little evidence that this results in a reduction in the incidence of catastrophic errors – the infrequent nature of these events makes this difficult, if not impossible, to prove 3. Despite this, summative assessment of individual non-technical skills is a requirement for re-licencing of pilots 4, and CRM training that includes team assessment of non-technical skills has been mandatory for flight crews in the UK since 1995. There is some evidence that non-technical skills are at least as important in the context of the operating theatre as in the aircraft cockpit 5. There is also a modest but growing body of evidence that training in such skills can enhance patient safety 6, and in the more specific context of the operating theatre, a lack of non-technical skills has been shown to be associated with poor surgical outcomes 7. However, applying lessons from the aviation industry to the operating theatre has proved difficult. The operating theatre is a much more complex and less constrained environment than the cockpit, with a wide variety of patients presenting unique technical challenges and even less consistency of staffing than air crews. There is also more variability in patterns of work, and with the involvement of multiple specialties, much more scope for complex social and cultural interactions. The concept that assessment drives learning is not new, and many regard assessment as an essential aspect of the learning process 8. Many assessment tools for non-technical skills have been developed over many years, and Professor Rhona Flin and other staff from the University of Aberdeen have been at the forefront of this process in the aviation industry and other high-risk environments. In aviation, they developed the NOTECHS (non-technical skills) system for assessing pilot skills 9. In the context of operating theatres, they have developed assessment tools for anaesthetists (ANTS) 10-13, surgeons (NOTSS) 14, and scrub practitioners (SPLINTS) 15, 16. In this issue of Anaesthesia, Rutherford et al. describe the development of ANTS-AP, a new tool for assessing the non-technical skills of anaesthetic practitioners, a group term that includes anaesthetic nurses and operating department practitioners 17. This paper provides useful insights into the challenges of developing an assessment tool suitable for summative assessment of non-technical skills. The process starts with the identification of behavioural markers linked to non-technical skills that are relevant to the workforce for which the assessment tool is developed. Skills can only be reliably assessed when they result in overt behaviours, and the first problem is that some non-technical skills (such as decision-making) are cognitive, and only become measurable when communicated. It can be deduced that a good assessment of non-technical skills requires consistent and insightful interpretation of behaviour, and this in turn requires the development of behavioural markers that are clear and consistent, and do not overlap with other behavioural markers. In the clinical environment, many non-technical skills are often only apparent in composite behaviours that may be difficult to separate and quantify. Although Rutherford et al. developed a taxonomy of behaviours specifically for the assessment of anaesthetic practitioners 17, the domains of behaviour identified are similar to those in other assessment tools, and to some extent the process was informed by the authors’ past experiences. Rutherford et al. used a Delphi technique, an iterative process that utilises the opinions of experts 18, to identify behavioural markers appropriate to the work of anaesthetic assistants. These were then coded and developed into domains of non-technical skills that were later assessed in a graded manner. Other methods of identifying behavioural markers use retrospective analysis of accidents or prospective observation of staff in the operating theatre environment 19. With a Delphi technique, the validity of the assessment depends on the ability of the expert group to identify accurately ‘good’ and ‘bad’ behaviours from the workplace that correlate with non-technical skills. The experts therefore have to be familiar with the workplace for which the assessment tool is developed, and ideally they should represent a breadth of opinion from a variety of working cultures. Rutherford et al. used both anaesthetists and anaesthetic practitioners in the expert group. Despite using experts with a variety of backgrounds and skills, there is always a subjective element to assessment that may only become apparent during later testing. Rutherford et al. describe how variations in the assessors’ workplace culture affected the interpretation of actions as either correct or incorrect at the assessment stage, leading to unplanned variation in scoring and reduced inter-rater reliability. This illustrates the difficulty of scoring behaviours, even after the relevant behavioural markers have been agreed following a robust process. To some extent, this effect can be reduced, but not completely eliminated, by assessor calibration. The assessments usually take place in a simulated theatre environment. Depending on the fidelity of the simulation, this has the potential to reduce assessment validity. The important aspect of validity here is construct validity, the extent to which the assessment is able to measure skills that are relevant in the workplace. There may a reciprocal relationship with inter-rater reliability, the ability of repeated assessments of the same behaviours to produce the same result. Simulated environments tend to use standardised scenarios that reduce variation in order to improve the reliability of the assessment process, categorising user actions as ‘right’ or ‘wrong’. Real life is rarely so simple, and a reductive approach may result in a simplistic assessment, with little relationship to real-world performance and poor validity. Real operating theatres are extremely complex social environments. Space is often limited, and staff may behave in a territorial fashion. There are multiple interactions between individuals, monitors, patients and healthcare workers, creating infinite sources of variation 20. Behaviour is affected by professional boundaries and personal factors such as perceived status, experience and culture, altering the power dynamic of relationships and potentially making errors more likely. These factors are extremely difficult to reproduce accurately in a simulated environment, and the best that can be achieved is a realistic compromise acceptable to both learners and assessors. The knowledge that all participants are working in a simulated environment that is constrained by practical considerations may in itself constrain team behaviours and individual responses to simulated scenarios. This adds to the difficulties for the assessor in interpreting behavioural markers. Actions of other ‘helper’ individuals are typically scripted, but the fidelity and validity of the simulation then depend on acting skills and accurate recall of scripted actions. Also, poor support from simulated team members is capable of impairing individual performance. It could be argued that this is what happens in real life, so that what is lost in terms of reliability is gained in validity! The ability of the assessors to record different types of non-technical skills accurately is also affected by time factors, distractions to the assessor and many other confounding variables. One of the ways to compensate for this is to videotape interactions, which solves the problem of the time taken to perform an assessment – but introduces other problems. As well as the potential for behaviour to be affected by the presence of the camera and the knowledge that performance is recorded, the position of the camera affects the perspective of the assessor, whose view may be intermittently obstructed by staff or equipment in addition. Assessment of communication and cognition may also be dependent on the quality of the audio recording, which in Rutherford et al.'s study was not always ideal despite the investigators’ best efforts. Using multiple video cameras and microphones may help with the quality of recording, but could also add to the complexity of the assessment and may make it more time-consuming. As Rutherford et al. have demonstrated, developing a valid and reliable individual assessment tool and validating it in a simulated environment is not easy. For a high-stakes process such as qualification or recertification, a summative assessment is appropriate and the test environment needs to be carefully designed to be able to produce good validity and reliability. In a formative assessment, the main aim is to produce quantified feedback to the learner that guides skills development. In this case, measures of reliability are not essential to justify the process, so the default position for many courses and organisations is to promote formative rather than summative assessment of non-technical skills. Given the difficulties with development and the care that has been invested in producing valid and reliable assessment tools such as ANTS, it is perhaps surprising that they are not used more commonly for summative assessments in anaesthesia 21 and other operating theatre specialities, in the same way that they are used in the aviation industry. If we believe that non-technical skills are as important in operating theatres as they are in the cockpit of a jet, then perhaps we should adopt the same standards and require re-certification of theatre staff in these skills. The practical challenges of this approach would certainly be formidable. Now that assessment tools for non-technical skills in other specialities have been developed, it will be interesting to see how they are used. With increasing political pressures on organisations to justify what they are doing to promote patient-centred and safety-conscious cultures, a recertification process that uses a summative assessment of an individual's non-technical skills may seem an attractive proposition. If these sophisticated assessment tools are not used for summative assessments of non-technical skills, what is the point of further development? Currently, the most cited use of these tools in the literature seems in be in educational research, to identify behavioural markers and quantify non-technical skills. If sufficient care is given to planning the simulated environment and training assessors, assessment tools such as ANTS provide useful ways to study and evaluate educational interventions. If it is decided that the practical problems of producing individual summative assessments cannot be justified, an alternative approach might be to focus on team training and formative assessment based on a CRM model. This has been extensively used in anaesthesia team training 22, and may be regarded as more proportionate way to promote non-technical skills than summative assessment of individuals. Again, there are similarities between air crews and theatre teams, where individual team members have consistent tasks and roles but may not always work with the same individuals. It could be argued that team training and assessment is better suited to the working environment, focusing on actual roles rather than simulated roles, adding authenticity to the training process. Because theatre teams have more complex interactions than air crews, reliable team assessments may be difficult to achieve 23, but there is evidence that CRM training can result in measurable improvements in the non-technical skills within teams if the practical challenges of transferring this type of training to the operating theatre environment can be overcome 24. There is also some evidence that cognitive aids may help to improve team performance 25, 26. Regular mandatory theatre team training could be a good way to provide reassurance to stakeholders that an organisation is doing all it can to promote teamworking and avoid errors, particularly if improvements in patient outcomes can be demonstrated as a result of the training. Whatever approach to non-technical skills training and assessment is taken, it should be remembered that skills deteriorate rapidly following training unless they can be successfully transferred to the working environment 27. This is particularly apparent in resuscitation training, where training for infrequent events will not be followed up by practice in the workplace, leading to significant deterioration in skills 28. However, unlike resuscitation skills, non-technical skills have the potential to be continuously reinforced in a supportive operating theatre environment. This could be expected to lead to reduced deterioration of skills following training, but there seems to be little supporting evidence for this. There seems to be a wealth of evidence available to support the promotion and development of non-technical skills in individuals and teams. Validated assessment tools are not currently being used to their full potential in the training and assessment of operating theatre staff, but if they were adopted for high-stakes assessments of operating theatre staff this would require a major cultural shift. There seems to be increasing pressure on healthcare organisations within the UK to reduce medical errors, and promoting the development of standardised training of non-technical skills in operating theatre staff would seem a logical next step. A recent retrospective analysis of airway incidents from the fourth National Audit Project (NAP4) has highlighted the continuing importance of non-technical skills in error causation in anaesthesia 29. The Royal College of Anaesthetists promotes non-technical skills during training and quotes the ANTS domains of behavioural markers in its latest curriculum guidance 30, but there does not seem to be any trend towards routine theatre team training, despite promotion as an Anaesthesia Clinical Services Accreditation (ACSA) standard 31. We seem to be lagging substantially behind other high-risk professions in this regard, and perhaps now would be a good time to think about how we could catch up. No external funding and no competing interests declared.