Assessment of Endovascular Team Performances Using a Comprehensive Data Capture Platform in the Hybrid Room: A Pilot Study

Abstract

The hybrid room (HR) is a complex, high risk environment, requiring teams (surgeons, anaesthetists, nurses, technicians) to master various skills. Routine (team) performance evaluation is uncommon. To minimise errors and prevent these from escalating or re-occurring, workplace based assessments could form a solution.1Lear R. Riga C. Godfrey A.D. Falaschetti E. Cheshire N.J. Van Herzeele I. et al.Multicentre observational study of surgical system failures in aortic procedures and their effect on patient outcomes.Br J Surg. 2016; 103: 1467-1475Crossref PubMed Scopus (18) Google Scholar The Operation Room Black Box (ORBB; Surgical Safety Technologies Inc. Toronto, Canada) is a medical data recording system that captures audiovisual and procedural data. It has been used to facilitate structured analysis of team performance during laparoscopic surgery.2Jung J.J. Jüni P. Lebovic G. Grantcharov T. First-year analysis of the Operating Room Black Box Study.Ann Surg. 2020; 271: 122-127Crossref PubMed Scopus (77) Google Scholar This pilot study aimed to establish whether this new data capturing technology could be used to analyse technical, non-technical, and radiation safety performances of teams in the HR during endovascular procedures and to evaluate environmental influences. A single centre observational study was conducted between February and June 2019 after the first ORBB installation in a HR.3Doyen B. Gordon L. Soenens G. Bacher K. Vlerick P. Vermassen F. et al.Introduction of a surgical Black Box system in a hybrid angiosuite: Challenges and opportunities.Phys Med. 2020; 76: 77-84Abstract Full Text Full Text PDF PubMed Scopus (2) Google Scholar Video footage of the fluoroscopy screen and anaesthesia monitor was combined with data from four ceiling mounted cameras and three ceiling array microphones. Elective endovascular infrarenal abdominal aorto-iliac aneurysm repair (EVAR) or treatment of symptomatic iliac, femoral, and popliteal atherosclerotic disease (PVI) were recorded. Procedures were captured only when patients and all team members had provided written informed consent. Team performance was assessed using validated measures and established frameworks (Table 1). When possible, (semi-automatic) evaluation techniques used in laparoscopic surgery were retained; otherwise, alternatives were chosen, based on previous experiences.4Doyen B. Maurel B. Hertault A. Vlerick P. Mastracci T. Van Herzeele I. Radiation safety performance is more than simply measuring doses! Development of a radiation safety rating scale.Cardiovasc Intervent Radiol. 2020; 43: 1331-1341Crossref PubMed Scopus (4) Google Scholar,5Van Herzeele I. Aggarwal R. Malik I. Gaines P. Hamady M. Darzi A. et al.Validation of video-based skill assessment in carotid artery stenting.Eur J Vasc Endovasc Surg. 2009; 38: 1-9Abstract Full Text Full Text PDF PubMed Scopus (45) Google Scholar Technical success was defined as the absence of type I/III endoleak in EVAR and successful revascularisation in PVI procedures.Table 1Frameworks and outcome measures of the assessment of endovascular team performances in the hybrid roomCategory and frameworks usedOutcome measuresMedian (IQR)Technical skills in EVAR (n = 6) GRS: Modified global rating scale of generic endovascular skillsTotal scale score /40; ‘Acceptable’ score: 24∗Represents an average rating of 3/5 on all rating scale items.31.5 (27.5–34) PRS: Procedure specific rating scale for technical performance in EVARTotal scale score /35; ‘Acceptable’ score: 21∗Represents an average rating of 3/5 on all rating scale items.28.5 (27–30.75) EVARATE: EndoVascular Aortic Repair Assessment of Technical ExpertiseTotal scale score /35; ‘Acceptable’ score: 21∗Represents an average rating of 3/5 on all rating scale items.30 (29–31.75)Technical skills in PVI (n = 16) GRS: Modified global rating scale of generic endovascular skillsTotal scale score /40; ‘Acceptable’ score: 24∗Represents an average rating of 3/5 on all rating scale items.32.5 (30–34.5) PRS: Examiner Checklist for diagnostic angiography, angioplasty and stentingTotal scale score /85; ‘Acceptable’ score: 51∗Represents an average rating of 3/5 on all rating scale items.82 (81–84)Non-technical skills of surgical team NOTSS: Non-Technical Skills for SurgeonOverall scores scale categories (range 1–4)‡Non-Technical Skills NOTSS and SPLINTS scores: 1 = poor; 2 = marginal; 3 = acceptable; 4 = good. Situational awareness3 (3–3) Decision making3.5 (3–4) Communication teamwork3 (3–3) Leadership3 (3–3)No. of negative/positive behaviours per scale elementNon-technical skills of nursing team SPLINTS: Scrub Practitioners’ List of Intra-operative Non-Technical SkillsOverall scores scale categories (range 1–4)‡Non-Technical Skills NOTSS and SPLINTS scores: 1 = poor; 2 = marginal; 3 = acceptable; 4 = good. Situational awareness3 (3–3) Communication teamwork3 (3–4) Task management3 (2–3)No. of negative/positive behaviours per scale elementRadiation safety practices Dose measurements†Semi-automated analysis; analysis of other elements requires a significant amount of human input.Fluoroscopy time, Dose area product, Air Kerma Safety behaviour: increasing distancePosition of team members during DSA runsDistractions DiSI: Modified Disruptions in Surgery Index†Semi-automated analysis; analysis of other elements requires a significant amount of human input.Auditory distractions, No. in the room, No. of times door opens.IQR = interquartile range; GRS = global rating scale; PRS = procedure specific rating scale; EVAR = endovascular aneurysm repair; PVI = peripheral vascular intervention; DSA = digital subtraction angiography.∗ Represents an average rating of 3/5 on all rating scale items.† Semi-automated analysis; analysis of other elements requires a significant amount of human input.‡ Non-Technical Skills NOTSS and SPLINTS scores: 1 = poor; 2 = marginal; 3 = acceptable; 4 = good. Open table in a new tab IQR = interquartile range; GRS = global rating scale; PRS = procedure specific rating scale; EVAR = endovascular aneurysm repair; PVI = peripheral vascular intervention; DSA = digital subtraction angiography. Six EVAR and 16 PVI procedures were included. Forty-three unique team members participated, with a median team size of six (IQR 6, 7). The ORBB successfully recorded, transmitted, and synchronised the surgical phase. Radiation safety practices were successfully analysed. In 11 procedures, non-technical performance was rated “2-Marginal” (EVAR: 2; PVI: 4) or “1-Poor” (EVAR: 3; PVI: 2) for at least one Non-Technical Skills for Surgeons / Scrub Practitioners‟ List of Intraoperative Non-Technical Skill (NOTSS/SPLINTS) scale element. In one EVAR, surgeons’ communication and teamwork was rated “1-Poor” because of suboptimal communication with the anaesthetic team, that is confusion about heparin administration and when to start/stop apnoea during angiograms. “1-Poor” ratings were also observed for NOTSS “leadership” once, and SPLINTS “task management” in two cases. Surgeons’ “decision making” was the highest rated NOTSS element (12/22 rated “4-Good”), because of exemplary shared decision making (e.g., whether to stent a lesion based on DSA). For nurses, the highest scores were observed in the SPLINTS “communication and teamwork” categories. Doors opened with a median frequency of 43 (IQR 34, 52) times per hour. Per procedure, a median of 12 (IQR 6, 23) auditory distractions was registered, originating from the anaesthetic machine (median 5.5 [IQR 2, 15]), C arm system (median 2 [IQR 1, 3]), and phone calls (median 2.5 [IQR 1, 4]). Technical success was achieved in all EVARs, but technical performance was once rated poor (PRS: 19/35). In four EVAR (66%) and three (18.8%) PVI procedures, the supervisor took over. PVI revascularisation was unsuccessful in one patient because of vessel spasm, which resolved after papaverine administration. At 30 days, re-intervention was needed because of rest pain. In another case, a flow limiting dissection was observed and left untreated, with an acceptable 30 day clinical outcome. Technical performances during these procedures were among the lowest rated. This observational study is the first describing the use of an ORBB in a HR and demonstrated the feasibility of using ORBB driven workplace based assessment of technical and non-technical skills, radiation safety practices, and environmental distractions/disruptions. Besides detecting areas for improvement, the ORBB may also highlight positive team behaviour. This single centre study included only a small sample of diverse procedures and a Hawthorne effect cannot be excluded, meaning that team members’ performances may be affected by knowing that they were being assessed. Despite successful use in laparoscopy, different procedural types and the novel HR setting required introduction of new frameworks to assess team performances (Table 1).2Jung J.J. Jüni P. Lebovic G. Grantcharov T. First-year analysis of the Operating Room Black Box Study.Ann Surg. 2020; 271: 122-127Crossref PubMed Scopus (77) Google Scholar,4Doyen B. Maurel B. Hertault A. Vlerick P. Mastracci T. Van Herzeele I. Radiation safety performance is more than simply measuring doses! Development of a radiation safety rating scale.Cardiovasc Intervent Radiol. 2020; 43: 1331-1341Crossref PubMed Scopus (4) Google Scholar Although analysis currently still requires a lot of human input, the triangulation of video based behavioural observations, rating scale based assessments, and objective performance indicators provides a comprehensive analysis of team performance, which may help to optimise team functioning and performance, and augment patient and overall safety by learning from mistakes/errors. Next, a large scale prospective baseline study will closely analyse team performance in approximately 100 elective endovascular procedures to catalogue near misses and errors in the HR and carry out a root cause analysis. This baseline will foster well powered interventional studies in the future. Targeted quality improvement initiatives will be implemented, for example every team member will have access to a massive open online course to improve radiation safety practices, which combines E learning, video scenarios, and game based learning. After implementation, the ORBB will allow evaluation of the impact on near misses, errors, and radiation safety practices of the team. Furthermore, ORBB can monitor safety processes such as the WHO surgical safety checklist and drive psychological research on how teams should ideally function (e.g., leadership qualities, team communication types). ORBB based assessment may also facilitate structured multidisciplinary debriefing and provide feedback. This paper won first prize as a digital abstract at the ESVS Month 2020. Teodor Grantcharov: has IP ownership and a leadership role in Surgical Safety Technologies Inc.