Non-Technical Skills For Surgeons Research Articles

WP8.2 - Human Factors Analysis of CORESS cases using the NOTSS Framework

Abstract Aims Determine human factors implicated in cases submitted to the Confidential Reporting System in Surgery (CORESS), using the non-technical skills for surgeons (NOTSS) framework. Methods Consecutive cases submitted to CORESS, between 2005 and 2011 were evaluated as an interim analysis. The type of case, whether patient harm occurred, Clavien-Dindo (CD) classification, and the location of the incident were collected. Demographic data was not collected because they are often changed to protect patient identification. Each case was reviewed according to the non-technical skills for surgeons (NOTSS) taxonomy. Results There were 100 cases included in this interim analysis. The most common types of cases were delayed diagnosis (n=19), known complications (n=10), near misses (n=10), poor surgical technique (n=9), misuse of equipment/device (n=8), and wrong site surgery (n=5). Two-thirds of patients were harmed, with complications ranging from minor to life-threatening (9 CD I, 10 CD II, 32 CD III, 12 CD IV), and 4 resulting in death (CD V). Incidents occurred most frequently in the operating theatre (n=54), the ward (n=16), or the emergency department (n=9). There were 71 incidents that endangered patient safety related to deficiencies in situational awareness, 59 to poor communication and teamwork, 48 to poor decision-making, and 48 related to leadership failures. Conclusions The main human factors implicated in surgical patient safety incidents related to errors in situational awareness, communication and teamwork, and to a lesser extent, decision-making, and leadership. Efforts to prevent medical errors should be focused on clinician human factors training.

Nontechnical Skills Assessment in Acute Care Trauma Simulations: A Mixed Methods Approach Using Eye Tracking and Behavioral Marker Systems.

The condition of trauma patients and the urgent need for timely resuscitation present unique challenges to trauma teams. These difficulties are exacerbated for military trauma teams in combat environments. Consequently, there is a need for continued improvement of nontechnical skills (NTS) training for trauma teams. However, current approaches to NTS assessment rely on subjective ratings, which can introduce bias. Accordingly, objective methods of NTS evaluation are needed. Eye-tracking (ET) methods have been applied to studying communication, situation awareness, and leadership in many health care settings, and could be applied to studying physicians' NTS during trauma situations. In this study, we aimed to assess the relationship between trauma team leaders' objective gaze patterns and subjective expert NTS ratings during patient care simulations. After Institutional Review Board approval, 9 trauma teams from first-year post-graduate general surgery and emergency medicine residents were recruited to participate in 1 of 2 trauma simulations (a difficult airway case and a multi-patient trauma). Each scenario lasted approximately 15 minutes. All team leaders wore a mobile ET system to evaluate gaze metrics-time to first fixation (TTFF), average fixation duration (AFD), and total percentage of the scenario (TPS) focused on Areas of Interest (AOI), which included patient, care team, diagnostic equipment, and patient care equipment. Trained faculty raters completed the Non-Technical Skills for Surgeons (NOTSS) assessment tool and the Trauma Non-Technical Skills (T-NOTECHS) scale. One-way analysis of variance, Kruskal-Wallis, and appropriate post-hoc pairwise comparison tests were run to assess differences between ET metrics across AOI groups. Spearman's Rho tests were used to assess correlations between ET and subjective NTS ratings. Compared to other NTS domains, trauma teams scored relatively poorly on communication across both T-NOTECHS (3.29$ \pm $0.61, maximum = 5) and NOTSS (2.87$ \pm $0.66, maximum = 4). We found significant differences in trauma team leaders' TTFF between teammates and the patient (Team: 1.56 vs Patient: 29.82 seconds, P < .001). TTFF on the diagnostic equipment was negatively correlated (P < .05) to multiple measures of subjective NTS assessments. There were no significant differences in AFD between AOIs, and AFD on teammates was positively correlated (P < .05) to communication and teamwork. There were significant differences in TPS across most AOI pairs (P < .05), and the average TPS fixated was highest on the patient (32%). Finally, there were several significant correlations between additional ET and NTS metrics. This study utilized a mixed methods approach to assess trauma team leaders' NTS in simulated acute care trauma simulations. Our results provide several objective insights into trauma team leaders' NTS behaviors during patient care simulations. Such objective insights provide a more comprehensive understanding of NTS behaviors and can be leveraged to guide NTS training of trauma physicians in the future. More studies are needed to apply these methods to capture NTS from a larger sample of teams in both simulated and real trauma environments.

Deep Learning Analysis of Surgical Video Recordings to Assess Nontechnical Skills

Assessing nontechnical skills in operating rooms (ORs) is crucial for enhancing surgical performance and patient safety. However, automated and real-time evaluation of these skills remains challenging. To explore the feasibility of using motion features extracted from surgical video recordings to automatically assess nontechnical skills during cardiac surgical procedures. This cross-sectional study used video recordings of cardiac surgical procedures at a tertiary academic US hospital collected from January 2021 through May 2022. The OpenPose library was used to analyze videos to extract body pose estimations of team members and compute various team motion features. The Non-Technical Skills for Surgeons (NOTSS) assessment tool was employed for rating the OR team's nontechnical skills by 3 expert raters. NOTSS overall score, with motion features extracted from surgical videos as measures. A total of 30 complete cardiac surgery procedures were included: 26 (86.6%) were on-pump coronary artery bypass graft procedures and 4 (13.4%) were aortic valve replacement or repair procedures. All patients were male, and the mean (SD) age was 72 (6.3) years. All surgical teams were composed of 4 key roles (attending surgeon, attending anesthesiologist, primary perfusionist, and scrub nurse) with additional supporting roles. NOTSS scores correlated significantly with trajectory (r = 0.51, P = .005), acceleration (r = 0.48, P = .008), and entropy (r = -0.52, P = .004) of team displacement. Multiple linear regression, adjusted for patient factors, showed average team trajectory (adjusted R2 = 0.335; coefficient, 10.51 [95% CI, 8.81-12.21]; P = .004) and team displacement entropy (adjusted R2 = 0.304; coefficient, -12.64 [95% CI, -20.54 to -4.74]; P = .003) were associated with NOTSS scores. This study suggests a significant link between OR team movements and nontechnical skills ratings by NOTSS during cardiac surgical procedures, suggesting automated surgical video analysis could enhance nontechnical skills assessment. Further investigation across different hospitals and specialties is necessary to validate these findings.

Unveilling the hidden skillset: exploring non-technical skills in surgical education across spanish medical universities.

Non-Technical Skills (NTS) are cognitive, social, and personal resource skills that are crucial in complex and high-risk environments. The aims of our research are to determine the prevalence and content of NTS in the surgical rotation teaching guides of the Medicine Degree programs in Spanish Universities, to identify the most prevalent types and subtypes of NTS, and to analyze factors associated with the prevalence of surgical NTS in Medical Schools in Spain. Descriptive observational cross-sectional study involving the identification and collection of competencies outlined in the surgical rotation teaching guides of Spanish Medical Schools. Information regarding university performance was obtained from the Foundation for Knowledge and Development Ranking webpage. The "Non-Technical Skills for Surgeons" (NOTSS) system was used to classify each competency in the teaching guides as NTS (categories and elements) and technical skills. Disagreements were resolved through group consensus. A total of 1,846 competencies were analyzed in surgical rotations of the Medicine Degree programs across 40 Spanish Universities, with 99 competencies identified as surgical NTS, accounting for 5% of the total. The most frequently identified surgical NTS were "Decision Making" (46%), "Communication & Teamwork" (25%), and "Leadership" (19%). Additionally, several NOTSS were not identified in any institution. Public universities and those including a greater number of competencies had a higher rate of surgical NTS competencies, and we did not find a correlation between surgical NTS competencies and quality indices of University Centers. There is a limited presence of surgical NTS in the educational plans of Spanish Universities.

Simulation-based training for intraoperative posterior capsule rupture management: an analysis of nontechnical skills development and the relationship between technical and nontechnical skills.

To analyse development of individual nontechnical skills (NTS) domains after undertaking a previously developed simulation-based training model and analyse the relationship between technical skills (TS) and NTS in ophthalmic surgery. The simulation-based training model involved a cataract surgery case complicated by intraoperative posterior capsule rupture. Cataract surgeons underwent the simulation twice, separated by a training intervention. Two blinded independent experts assessed participants' NTS using HUFOES, NOn-Technical Skills for Surgeons (NOTSS), and the OSATS global rating scale for TS. Paired t-tests assessed differences in individual NTS domains, with p < 0.05 indicating significance. The Pearson Product Moment Correlation Coefficient was used to assess the correlation between scores from each scoring system. All NTS domains within HUFOES and NOTSS demonstrated statistically significant improvements secondary to the training intervention. Positive correlations were demonstrated between HUFOES and OSATS scores in the pre- and post-training simulations, r = 0.870 (p < 0.001) and r = 0.861 (p < 0.001), respectively. Positive correlations were also demonstrated between NOTSS and OSATS scores in pre- and post-training simulations, r = 0.849 (p < 0.001) and r = 0.757 (p = 0.001), respectively. Positive correlations were demonstrated between HUFOES and NOTSS scores; r = 0.979 (p < 0.001) (n = 17) and r = 0.959 (p < 0.001) for pre- and post-training simulations, respectively. All NTS domains contained within HUFOES and NOTSS demonstrated significant increases following the completion of the simulation-based training model. Positive correlations exist between an ophthalmic surgeon's TS and NTS. This is the first study to report these findings within ophthalmic surgery.

The making of a (new) surgeon: Opportunities for repeated messaging on NOTSS in the Canadian medical system

Proficiency in nontechnical skills (NTS) contributes to reduction in critical safety incidents and improvement in patient safety outcomes. Despite evidence demonstrating the importance of NTS in patient safety, there remains limited NTS specific curricula and formal teaching in Canadian surgical programs. We propose a three-stage longitudinal approach to education surrounding NTS using the Nontechnical skills for surgeons (NOTSS) framework.

1132 UseBeforeSurgery-Practical Tips (UBS-PT) Initiative: Enhance Patient Safety with Tips Inspired by Military and Civil Aviation

Abstract Aim The healthcare system, to improve its culture of safety, often have looked at high-reliability organisations including the aviation industry. However, while some flight-safety tools have been slowly introduced into perioperative procedures (e.g., Surgical-Safety Checklist), many guidelines related to Non-Technical-Skills for Surgeons (NOTSS) have obtained low adherence. The apparent time-consuming nature of their implementation have represented a major barrier to the flourish of the NOTSS system. Thus, complementary initiatives, such as the UBS-PT presented here, are needed. The UBS-PT is a list of practical tips aimed at improving patient safety and surgical performance. Method By applying a reiterative evidence-based developing process involving healthcare professionals, military pilots, human factors, and legal experts, we identified several practical tips to enhance surgical performance. The developing process and operationalization were complemented by a deep literature review to weight tips’ efficiency, cost-effectiveness, usability, and ease of implementation. Results Within NOTSS system framework, we defined ten practical tips promoting the use of (1) group chair-flying, (2) colour-coded stickers, (3) standardised layouts, (4) closed-loop communication, (5) labelled and colour-coded surgical caps, (6) cross-checking and (7) two-challenge rule procedures, (8) the sterile cockpit rule, (9) self-checklist, and (l0) tactical napping. Conclusions In an attempt to meet the increasing standards of quality and safety in patient care, the UBS-PT initiative might encourage surgical teams and patient safety boards to find more suitable initiatives for each specific context. The UBS-PT initiative might represent the first step towards a tailored approach, offering concrete measures for the development of safety practices at individual and team levels.

Non-technical skills for surgeons (NOTSS)

Non-technical skills for surgeons (NOTSS)

Nontechnical Skills for Surgeons as a Framework to Evaluate Cardiopulmonary Bypass Management Skills of Resident Trainees

Nontechnical skills are critical in cardiac surgery but currently there is no formal paradigm to teach these in residency training. We investigated the use of the Nontechnical skills for surgeons (NOTSS) system as a framework to assess and teach nontechnical skills related to cardiopulmonary bypass (CPB) management. Single-center retrospective analysis of Integrated and Independent pathway thoracic surgery residents who participated in dedicated nontechnical skills evaluation and training. Two CPB management simulation scenarios were utilized. All residents received a lecture on CPB fundamentals and then individually participated in the first simulation ("Pre-NOTSS"). Immediately following this, nontechnical skills were rated by self-assessment and by a NOTSS trainer. All residents then underwent group NOTSS training followed by the second individual simulation ("Post-NOTSS"). Nontechnical skills were rated as before. NOTSS categories assessed included Situation Awareness, Decision Making, Communication and Teamwork, and Leadership. Nine residents were divided into 2 groups: Junior (n = 4, PGY1-4) and Senior (n = 5, PGY5-8). Pre-NOTSS resident self-ratings were higher for Senior than Junior in the categories of Decision Making, Communication and Teamwork, and Leadership while trainer ratings were similar between the groups. Post-NOTSS, resident self-ratings were higher for Senior than Junior in Situation Awareness and Decision Making while trainer scores were higher for both groups in Communication and Teamwork and Leadership. The NOTSS framework in conjunction with simulation scenarios provides a practical framework to evaluate and teach nontechnical skills related to CPB management. NOTSS training can lead to improvements in both subjective and objective ratings of nontechnical skills for all PGY levels.

Utilizing Eye Tracking to Assess Medical Student Non-Technical Performance During Scenario-Based Simulation: Results of a Pilot Study.

Non-technical skills (NTS) are essential for safe surgical patient management. However, assessing NTS involves observer-based ratings, which can introduce bias. Eye tracking (ET) has been proposed as an effective method to capture NTS. The purpose of the current study was to determine if ET metrics are associated with NTS performance. Participants wore a mobile ET system and participated in two patient care simulations, where they managed a deteriorating patient. The scenarios featured several challenges to leadership, which were evaluated using a 4-point Likert scale. NTS were evaluated by trained raters using the Non-Technical Skills for Surgeons (NOTSS) scale. ET metrics included percentage of fixations and visits on areas of interest. Ten medical students participated. Average visit duration on the patient was negatively correlated with participants' communication and leadership. Average visit duration on the patient's intravenous access was negatively correlated with participants' decision making and situation awareness. Our preliminary data suggests that visual attention on the patient was negatively associated with NTS and may indicate poor comprehension of the patient's status due to heightened cognitive load. In future work, researchers and educators should consider using ET to objectively evaluate and provide feedback on their NTS.

Managing a team in the operating room: The science of teamwork and non-technical skills for surgeons

Managing a team in the operating room: The science of teamwork and non-technical skills for surgeons

Intraoperative Assessment of Non-Technical Skills for Surgeons (NOTSS) and Qualitative Description of their Effects on Intraoperative Performance

The aims of this study were (1) to evaluate the feasibility of using the Non-Technical Skills for Surgeons (NOTSS) rating tool in assessing surgeons' non-technical skills behaviors in live operations, and (2) to describe the effect of NOTSS on intraoperative performance. This study was conducted in an academic hospital in North America. Two observers independently conducted direct non-participant observations using the NOTSS rating tool to assess non-technical skills, and to document examples of effective or ineffective non-technical skills behaviors. Observers took field notes to document non-technical skill gaps that were not captured by the NOTSS rating tool, and situations or scenarios that presented challenges for accurate assessment. Interclass correlation estimates and 95% confidence intervals were calculated to assess the validity of the NOTSS rating tool. Deductive thematic qualitative data analysis was used for field notes and NOTSS behavior descriptions. Participants were general surgeons performing either minimally invasive (robotic assisted or laparoscopic surgery), or open procedures. We observed 18 surgeries, involving 6 surgeons, 11 residents and one fellow resulting in 37 hours of direct observations. The mean NOTSS score was 3.8 (SD 0.41) for situation awareness, 3.75 (SD 0.47) for decision-making, 3.71 (SD 0.39) for communication and teamwork, and 3.76 (SD 0.38) for leadership. The inter-rater reliability ranged between 0.65 and 0.80 for each NOTSS categories. The observers documented examples of effective non-technical skills behaviors and examples of behaviors that need improvement. Furthermore, we described the effect of each observed behavior on intraoperative performance. One challenge to NOTSS use in live surgery was that observers had to infer situation awareness, decision-making, and coping with pressure as these were not easily observed without attending surgeons articulating their underlying thought process. The use of the NOTSS tool in live surgery is a valid and practical tool to document observed behaviors and their effect on intraoperative performance in order to provide constructive feedback to surgeons. One notable limitation is that without specific articulation by the surgeon of their underlying thought process the observer must infer specific elements. By documenting specific real-world events with high inter-rater reliability and adequate surgeon score variation the process can be used to provide useful feedback for improvement.

Objective Nontechnical Skills Measurement Using Sensor-based Behavior Metrics in Surgical Teams.

The purpose of this study was to identify objective measures that predict surgeon nontechnical skills (NTS) during surgery. NTS are cognitive and social skills that impact operative performance and patient outcomes. Current methods for NTS assessment in surgery rely on observation-based tools to rate intraoperative behavior. These tools are resource intensive (e.g., time for observation or manual labeling) to perform; therefore, more efficient approaches are needed. Thirty-four robotic-assisted surgeries were observed. Proximity sensors were placed on the surgical team and voice recorders were placed on the surgeon. Surgeon NTS was assessed by trained observers using the NonTechnical Skills for Surgeons (NOTSS) tool. NTS behavior metrics from the sensors included communication, speech, and proximity features. The metrics were used to develop mixed effect models to predict NOTSS score and in machine learning classifiers to distinguish between exemplar NTS scores (highest NOTSS score) and non-exemplar scores. NTS metrics were collected from 16 nurses, 12 assistants, 11 anesthesiologists, and four surgeons. Nineteen behavior features and overall NOTSS score were significantly correlated (12 communication features, two speech features, five proximity features). The random forest classifier achieved the highest accuracy of 70% (80% F1 score) to predict exemplar NTS score. Sensor-based measures of communication, speech, and proximity can potentially predict NOTSS scores of surgeons during robotic-assisted surgery. These sensing-based approaches can be utilized for further reducing resource costs of NTS and team performance assessment in surgical environments. Sensor-based assessment of operative teams' behaviors can lead to objective, real-time NTS measurement.

Non-technical skills amongst multidisciplinary surgeons (notss) in a tertiary hospital in India

Non-technical skills for surgeons are a set of skills that are non-surgical in nature but make the surgical procedure safer for the patient. In this context, this is a study in a tertiary hospital where the non-technical skills of multi-disciplinary surgeons were observed and assessed when they were performing live surgeries. We found that none of our participant surgeons were rated poor for any skills, which endangered performance or potentially endangered patient safety and for which serious remediation would have been required. "Acceptable" standards of "Situational Awareness" and "Leadership" and "Good" standards of "Decision Making" and "Communication &amp; Teamwork", which were the four skills observed.

Learning NOTSS While Tying Knots: Integrating the Nontechnical Skills for Surgeons Course With Technical Surgical Skills Training, A Mixed Methods Study.

The aims of this study were to describe the process of integrating 2 established training programs, Nontechnical skills for surgeons, and a traditional essential surgical skills course and to measure the impact of this integrated course on the behaviors of interprofessional surgical teams in Rwandan district hospitals. Surgical errors and resulting adverse events are due to variability in both technical and nontechnical surgical skills. Providing technical and nontechnical skills training to the perioperative team may enhance the learning of both of these skills and promote safe intraoperative patient care. A quality improvement framework guided the process of integrating essential surgical skills and nontechnical skills into a single training program for surgical teams. The resulting 2-day training program was delivered to 68-person teams from 17 hospitals. Nontechnical skills for surgeons was taught through didactics and in the operating room, where preoperative briefing, intraoperative interactions, and postoperative debriefing were used as essential and nontechnical skills teaching moments. Postcourse surveys, follow-up interviews, focus groups, and direct observation of participants in the operating room were conducted to assess how participants implemented the knowledge and skills from the training into practice. Ninety-seven percent of the participants reported that they were satisfied with the course. Follow-up participant interviews and focus groups reported that the course helped them to improve their preoperative planning, intraoperative communication, decision-making, and postoperative debriefing. It is possible to implement an integrated essential surgical skills and nontechnical skills training course. Integrating nontechnical skills into essential surgical skills courses may enhance learning of these skills.

82 Surgeons Don’t Just Operate: Improving Junior Surgical Practitioners’ Non-Technical Skills Outside of the Operating Theatre Using High-Fidelity Simulation

The non-technical skills of surgeons play a significant role in patient confidence, experience and safety The aim of the study was to improve non-technical skills and confidence of junior surgical practitioners regarding DNACPR conversations and obtaining operative consent.We have developed a speciality-specific course to address the educational deficit surrounding non-technical skills outside of the operating theatre, focussing on DNACPR and consent, for junior surgical practitioners. It was piloted at Aintree University Hospital with their Orthopaedic Department. We combined lectures and workshops delivered by subject matter specialists with simulated scenarios. The simulated scenarios were debriefed by human factors specialists using the gold standard validated NOTSS tool The pilot course in June was a resounding success; candidate confidence scores increased significantly across both skills and knowledge surrounding capacity, consent and DNACPR. All attending candidates have recommended the course to their peers and suggested integration into the Foundation training curriculum. However, identifying and implementing courses appropriate for all foundation doctors with such a specialized course would be challenging. Most foundation doctors have at least one placement in a surgical specialty. Therefore, it is more realistic to adjust the course to become less specialized, encompassing all the surgical specialities, to target all foundation doctors initially instead of juniors in each speciality. Following the capture of foundation doctors, the second phase of implementation will return to speciality-specific courses to address the learning needs of non-medical practitioners, such as Physician Associates and Advanced Nurse Practitioners, working in surgical departments.

Transoral endoscopic thyroidectomy by a vestibular approach: cadaver simulation experience and ethicolegal issues.

This article describes the use of transoral endoscopic thyroidectomy by a vestibular approach on cadavers to improve the technical and non-technical skills of surgeons.

Strategies for Improving Quality and Safety in Global Health: Lessons From Nontechnical Skills for Surgery Implementation in Rwanda.

<h3>Key Messages</h3> Efforts to increase access to surgical care will achieve improved health outcomes only if those efforts are intertwined with efforts to increase surgical safety and quality. The World Health Organization Surgical Safety Checklist and Non-Technical Skills for Surgeons (NOTSS) framework are 2 tools to increase surgery quality and safety. The NOTSS for variable resource contexts is a new 1-day educational course developed in Rwanda that integrates contextually appropriate behaviors and values and can be implemented with low costs in any health care context. Program managers should provide context-specific NOTSS training for surgeons and operating theater staff. Policy makers should implement NOTSS framework into health care policy focusing on modern virtual teaching methods. Surgeons and operating theater staff should be familiar with the NOTSS framework and regularly update their knowledge through didactic courses, simulation, and online trainings.

BEST of Surgical Training: the pan-London Core Surgical Training induction programme: “There’s no I in team, but there is a me.”

BackgroundIn the UK, Core Surgical Training (CST) marks the start of a surgical career, but previous experience and skills vary widely. Whilst Individual hospital Trusts offer local inductions, these are generally of a generic, administrative nature rather than advising trainees on how best to harness training opportunities. We designed a regional induction programme, ‘Building Excellence in Surgical Training’ (BEST) to address this, develop essential technical and non-technical skills, and engender support networks. MethodsAll incoming London Core Surgical Trainees (annual cohort size 90) were invited to participate, during the week prior to commencement of training. Trainees undertook 3 modules (portfolio, surgical skills and human factors-based simulation) and a research paper presentation day. We collected qualitative and quantitative data through a structured evaluation form, pre and post course Likert-scale scores and self-assessment utilising the non-technical skills for surgeons (NOTSS) framework. Results972 CSTs have completed BEST over the past 12 years. In 2019, significant improvements were seen in: confidence for starting CST, 45% (n = 22/49)-83% (n = 33/40,p = 0.00045); feeling the core programme cared about them, 55% (n = 27/49)-98% (n = 41/42, p =< 0.00001); getting to know peers 16% (n = 8/49)-88% (n = 35/40, p =< 0.00001); understanding human factors 82% (n = 40/49)-95% (n = 36/38, p = 0.00427); gaining confidence of trainers, 49% (n = 23/47)-86% (n = 31/36, p = 0.00114), and ability to speak out over patient safety concerns, 78% (n = 38/49)-97% (n = 37/38,p = 0.00019). NOTSS assessments showed significant improvement across all 4 criteria (n = 142, p =< 0.00001). ConclusionBEST equips trainees with the fundamental skills and confidence to safely embark on surgical training and provides tools to navigate the challenges training presents. The ethos of collaboration and support will aid the development of more resilient and empowered surgeons, vital in this era.

Non-technical skill assessments across levels of US surgical training

BackgroundTo ensure safe patient care, regulatory bodies worldwide have incorporated non-technical skills proficiency in core competencies for graduation from surgical residency. We describe normative data on non-technical skill ratings of surgical residents across training levels using the US-adapted Non-Technical Skills for Surgeons (NOTSS-US) assessment tool. MethodsWe undertook an exploratory, prospective cohort study of 32 residents—interns (postgraduate year 1), junior residents (postgraduate years 2–3), and senior residents (postgraduate years 4–5)—across 3 US academic surgery residency programs. Faculty went through online training to rate residents, directly observed residents while operating together, then submitted NOTSS-US ratings on specific resident’s intraoperative performance. Mean NOTSS-US ratings (total range 4–20, sum of category scores; situation awareness, decision-making, communication/teamwork, leadership each ranged 1–5, with 1=poor, 3=average, 5=excellent) were stratified by residents’ training level and adjusted for resident-, rater-, and case-level variables, using mixed-effects linear regression. ResultsFor 80 operations, the overall mean total NOTSS-US rating was 12.9 (standard deviation, 3.5). The adjusted mean total NOTSS-US rating was 16.0 for senior residents, 11.6 for junior residents, and 9.5 for interns. Adjusted differences for total NOTSS-US ratings were statistically significant across the following training levels: senior residents to interns (6.5; 95% confidence interval, 4.3–8.7; P < .001), senior to junior residents (4.4; 95% confidence interval, 2.5–6.2; P < .001), and junior residents to interns (2.1; 95% confidence interval, 0.3–3.9; P = .017). Differences in adjusted NOTSS-US ratings across residents’ training levels persisted for individual NOTSS-US behavior categories. ConclusionThese data and online training materials can support US residency programs in determining competency-based performance milestones to develop surgical trainees’ non-technical skills.