Objective Assessment Of Surgical Skills Research Articles

Skill assessment based on clutch use in cross-platform robot-assisted surgery.

Many studies have investigated the transfer of skills between laparoscopic and robot-assisted surgery (RAS). These studies have considered time, error, and clinical outcomes in the assessment of skill transfer. However, little is known about the specific operations of the surgeon. Clutch control use is an important skill in RAS. Therefore, the present study aimed to propose a novel objective algorithm based on computer vision that can automatically evaluate a surgeon's clutch use. Additionally, the study aimed to evaluate the correlation between clutch metrics and surgical skill on different surgical robot platforms. The robotic surgery training center of Wuhan University trained 30 laparoscopic surgeons as the study group between 2023 and 2024. Laparoscopic surgeons were trained by combining robotic simulator exercises and RAS animal experiments. During the training, video and hand movement data were collected. Hand movements identified by a skin-color model were combined with labeling information to classify clutch use. The metrics were validated on different robotic platforms (dv-Trainer, EDGE MP1000, Toumai™ MT1000, and DaVinci Xi system) and among surgeons with different surgical skill levels. On the robotic simulator, clutch accuracy in the expert group was significantly higher than in the study group for all tasks. No significant differences were observed in the number of clutches between the expert and study groups. In the RAS experiment, the number of clutches decreased significantly for both study and expert groups. The accuracy was maintained at a high level in the expert group but decreased rapidly in the study group. We proposed a new objective assessment of surgical skills, clutch use metrics, in cross-platform RAS. Additionally, we verified that the metrics significantly correlated with the surgical skill levels of the surgeons.

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review.

There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity. A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis. A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes. Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

Development and validation of an objective assessment of surgical skill in arthroscopic management of meniscal tear: A pilot study

Development and validation of an objective assessment of surgical skill in arthroscopic management of meniscal tear: A pilot study

Surgical tooltip motion metrics assessment using virtual marker: an objective approach to skill assessment for minimally invasive surgery.

Surgical skill assessment has primarily been performed using checklists or rating scales, which are prone to bias and subjectivity. To tackle this shortcoming, assessment of surgical tool motion can be implemented to objectively classify skill levels. Due to the challenges involved in motion tracking of surgical tooltips in minimally invasive surgeries, formerly used assessment approaches may not be feasible for real-world skill assessment. We proposed an assessment approach based on the virtual marker on surgical tooltips to derive the tooltip's 3D position and introduced a novel metric for surgical skill assessment. We obtained the 3D tooltip position based on markers placed on the tool handle. Then, we derived tooltip motion metrics to identify the metrics differentiating the skill levels for objective surgical skill assessment. We proposed a new tooltip motion metric, i.e., motion inconsistency, that can assess the skill level, and also can evaluate the stage of skill learning. In this study, peg transfer, dual transfer, and rubber band translocation tasks were included, and nine novices, five surgical residents and five attending general surgeons participated. Our analyses showed that tooltip path length (p [Formula: see text] 0.007) and path length along the instrument axis (p [Formula: see text] 0.014) differed across the three skill levels in all the tasks and decreased by skill level. Tooltip motion inconsistency showed significant differences among the three skill levels in the dual transfer (p [Formula: see text] 0.025) and the rubber band translocation tasks (p [Formula: see text] 0.021). Lastly, bimanual dexterity differed across the three skill levels in all the tasks (p [Formula: see text] 0.012) and increased by skill level. Depth perception ability (indicated by shorter tooltip path lengths along the instrument axis), bimanual dexterity, tooltip motion consistency, and economical tooltip movements (shorter tooltip path lengths) are related to surgical skill. Our findings can contribute to objective surgical skill assessment, reducing subjectivity, bias, and associated costs.

An explainable machine learning method for assessing surgical skill in liposuction surgery.

Surgical skill assessment has received growing interest in surgery training and quality control due to its essential role in competency assessment and trainee feedback. However, the current assessment methods rarely provide corresponding feedback guidance while giving ability evaluation. We aim to validate an explainable surgical skill assessment method that automatically evaluates the trainee performance of liposuction surgery and provides visual postoperative and real-time feedback. In this study, machine learning using a model-agnostic interpretable method based on stroke segmentation was introduced to objectively evaluate surgical skills. We evaluated the method on liposuction surgery datasets that consisted of motion and force data for classification tasks. Our classifier achieved optimistic accuracy in clinical and imitation liposuction surgery models, ranging from 89 to 94%. With the help of SHapley Additive exPlanations (SHAP), we deeply explore the potential rules of liposuction operation between surgeons with variant experiences and provide real-time feedback based on the ML model to surgeons with undesirable skills. Our results demonstrate the strong abilities of explainable machine learning methods in objective surgical skill assessment. We believe that the machine learning model based on interpretive methods proposed in this article can improve the evaluation and training of liposuction surgery and provide objective assessment and training guidance for other surgeries.

A level set approach to high fidelity interactive bone shaving for virtual training of surgical osteotomies.

Interactive multimodal surgical simulators are powerful tools that allow efficient and objective assessment of surgical skills. Bilateral Sagittal Split Osteotomy (BSSO) requires precise cutting of the mandible with a motorized saw that cannot afford a high margin of error. Surgeons rely on visual and haptic cues that are hard to train for through existing curricula without training directly on patients. In this paper, we present a new algorithm to achieve low-cost, precise sawing of the bone that is capable of providing realistic force feedback to the user at haptic rates (~1000Hz). Our method is centered around treating the bone surface as a level set that is evolved at interactive rates upon user's interaction with the virtual saw. The movement of the virtual saw itself is governed by a rigid body solver which is attached to the pose (6 degrees of freedom) of the haptic device via a mass-spring-damper which also supplies haptic force feedback. Our method allows parameterization of the erosion rate, saw speed, and bone density thus making it suitable to any material shaving application including dentistry (e.g., tooth milling) and a variety of surgical osteotomies.

Deep neural networks for the assessment of surgical skills: A systematic review

Surgical training in medical school residency programs has followed the apprenticeship model. The learning and assessment process is inherently subjective and time-consuming. Thus, there is a need for objective methods to assess surgical skills. Here, we use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to systematically survey the literature on the use of Deep Neural Networks for automated and objective surgical skill assessment, with a focus on kinematic data as putative markers of surgical competency. There is considerable recent interest in deep neural networks (DNNs) due to the availability of powerful algorithms, multiple datasets, some of which are publicly available, as well as efficient computational hardware to train and host them. We have reviewed 530 papers, of which we selected 25 for this systematic review. Based on this review, we concluded that DNNs are potent tools for automated, objective surgical skill assessment using both kinematic and video data. The field would benefit from large, publicly available, annotated datasets representing the surgical trainee and expert demographics and multimodal data beyond kinematics and videos.

The computerized objective assessment of surgical skills: Considerations for counting the number of movements.

Motion capture and analysis techniques are emerging in the surgical education and surgical education research literature as viable ways to augment the assessment of technical skills. In particular, these methods provide an opportunity to reveal objective information about the efficiency of surgical procedures, above and beyond the accuracy of procedural outcomes. One assessment that is very prevalent in the literature are counts of the number of movements a surgeon makes in completing a technical performance. In this commentary, the number of movements metric is explored from kinesiology and engineering perspectives; two disciplines that have contributed heavily to the development of rigorous motion analysis methods. Furthermore, the assumption that skill efficiency improves linearly as a learner progresses along the continuum of expertise is challenged. While movement efficiency does certainly improve, this assumption does not necessarily capture the way that learners flexibly prioritize particular aspects of performance in the intermediate stages of skill learning. By way of this commentary, important a priori decisions that should proceed effective motion capture and analysis are highlighted, a call for the standardization of procedures is made, and an opportunity to better understand the way that computerized movement analysis techniques may contribute (or be detrimental) to competency constructs in surgical education and assessment is realized.

Expert Surgeons Can Smoothly Control Robotic Tools With a Discrete Control Interface

Objective assessment of surgical skill is gaining traction in a number of specialty fields. In robot-assisted surgery in particular, the availability of data from the operating console and patient-side robot offers the potential to derive objective metrics of performance based on tool movement kinematics. While these techniques are becoming established in the laparoscopic domain, current assessment techniques for robotic endovascular surgery are based primarily on observation, checklists, and grading scales. This work presents an objective and quantitative means of measuring technical competence based on analysis of the kinematics of endovascular tool tip motions controlled with a robotic interface. We designed an experiment that recorded catheter tip movement from 21 subjects performing fundamental endovascular robotic navigation tasks on a physical model. Motion-based measures of smoothness (spectral arc length and number of submovements) were computed and tested for correlation with subjective scores from a global rating scale assessment tool that has been validated for use when performing manual catheterization. Results show that the smoothness metrics that produced significant correlations with the global rating scale for manual catheterization show similar correlations for robotic catheterization. This finding is notable, since with the robotic interface, tool tip motion is commanded discretely via a control button interface, while in manual procedures the tools are controlled through continuous movements of the surgeon's hands. Logistic regression analysis using a single motion metric was capable of classifying subjects by expertise with better than 90% accuracy. These objective and quantitative metrics that capture movement quality could be incorporated into future training protocols to provide detailed feedback on trainee performance.

Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery.

With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge. We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels. We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved competitive accuracies of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within a 1-3 second window, without needing an observation of entire training trial. This study highlights the potential of deep architectures for efficient online skill assessment in modern surgical training.

Medical education assessment

One of the most critical steps for medical education is the assessment. The assessment can be divided into short-term memory, long-term memory, and retrieval memory. If the student acquires all these memories steps, the surgical skill will be mind incorporate for years. As a healthcare provider, the medical community needs to transform training and learning to a valid and reliable activity. Most of the medical evaluations are subjective; an objective assessment is difficult but most desirable. The authors described some objective surgical skill assessment based on OSAST, dexterity and global rating scale. Moreover, we discussed the formative and summative assessment roles to the medical learning process.

Use Endoscope and Instrument and Pathway Relative Motion as Metric for Automated Objective Surgical Skill Assessment in Skull Base and Sinus Surgery

Use Endoscope and Instrument and Pathway Relative Motion as Metric for Automated Objective Surgical Skill Assessment in Skull Base and Sinus Surgery

Evaluation of Skill Assessment After Hysteroscopy Training Curriculum a designed

Background: To evaluate a standard latex hysteroscopy model training course on gynecologists’ surgical skills. Methods: 74 gynecologists who attended to hysteroscopy training courses of Training Centre, Tehran University, 2013 to 2015 were investigated. Theoretical information of trainees were obtained at the beginning, at the end and 6 months after the course by filling up a reliabile and valid questionnaire. and its subjective and objective impact on their surgical skills and satisfaction were evaluated. Results: 87.84% of participants were satisfied by the curriculum and for most of them, virtual uteruses were felt realistic. After 6 months, subject impact of curriculum was 77.03%. But, 60.81% of gynecologists hardly could perform hysteroscopy, which was mostly due to lack of technology (53.22%)(Harders, 2006 #11). Objective assessment of surgical skills was done for those who could perform hysteroscopy. Of them 58.7% had experienced a significant increase in their surgical skills (p=0.004). Correlation of subjective and objective impact of curriculum was low (r=0.43), (p=0.003). Conclusion: Hysteroscopy training program can improve gynecologists’ hysteroscopic skills. The only index which is significantly correlated with the subjective impact of these courses is the number of previous operations. Also in Iran, because of lacking hysteroscopy equipment in most hospitals, the effect of training courses decreases after a while. So we recommend to train hysteroscopy in all medical universities and equip all hospitals with hysteroscope. Also we suggest training courses for all gynecologists and repeted it every some years.

Anatomical Region Segmentation for Objective Surgical Skill Assessment with Operating Room Motion Data.

Background Most existing objective surgical motion analysis schemes are limited to structured surgical tasks or recognition of motion patterns for certain categories of surgeries. Analyzing instrument motion data with respect to anatomical structures can break the limit, and an anatomical region segmentation algorithm is required for the analysis. Methods An atlas was generated by manually segmenting the skull base into nine regions, including left/right anterior/posterior ethmoid sinuses, frontal sinus, left and right maxillary sinuses, nasal airway, and sphenoid sinus. These regions were selected based on anatomical and surgical significance in skull base and sinus surgery. Six features, including left and right eye center, nasofrontal beak, anterior tip of nasal spine, posterior edge of hard palate at midline, and clival body at foramen magnum, were used for alignment. The B-spline deformable registration was adapted to fine tune the registration, and bony boundaries were automatically extracted for final precision improvement. The resultant deformation field was applied to the atlas, and the motion data were clustered according to the deformed atlas. Results Eight maxillofacial computed tomography scans were used in experiments. One was manually segmented as the atlas. The others were segmented by the proposed method. Motion data were clustered into nine groups for every dataset and outliers were filtered. Conclusions The proposed algorithm improved the efficiency of motion data clustering and requires limited human interaction in the process. The anatomical region segmentations effectively filtered out the portion of motion data that are out of surgery sites and grouped them according to anatomical similarities.

Objective Assessment of Surgical Skills in a 2-Day Visceral Anastomosis Techniques Course hold in the Annual Congress of the German Surgical Society (DGCH)

Objective Assessment of Surgical Skills in a 2-Day Visceral Anastomosis Techniques Course hold in the Annual Congress of the German Surgical Society (DGCH)

Assessing Arthroscopic Skills Using Wireless Elbow-Worn Motion Sensors.

Assessment of surgical skill is a critical component of surgical training. Approaches to assessment remain predominantly subjective, although more objective measures such as Global Rating Scales are in use. This study aimed to validate the use of elbow-worn, wireless, miniaturized motion sensors to assess the technical skill of trainees performing arthroscopic procedures in a simulated environment. Thirty participants were divided into three groups on the basis of their surgical experience: novices (n = 15), intermediates (n = 10), and experts (n = 5). All participants performed three standardized tasks on an arthroscopic virtual reality simulator while wearing wireless wrist and elbow motion sensors. Video output was recorded and a validated Global Rating Scale was used to assess performance; dexterity metrics were recorded from the simulator. Finally, live motion data were recorded via Bluetooth from the wireless wrist and elbow motion sensors and custom algorithms produced an arthroscopic performance score. Construct validity was demonstrated for all tasks, with Global Rating Scale scores and virtual reality output metrics showing significant differences between novices, intermediates, and experts (p < 0.001). The correlation of the virtual reality path length to the number of hand movements calculated from the wireless sensors was very high (p < 0.001). A comparison of the arthroscopic performance score levels with virtual reality output metrics also showed highly significant differences (p < 0.01). Comparisons of the arthroscopic performance score levels with the Global Rating Scale scores showed strong and highly significant correlations (p < 0.001) for both sensor locations, but those of the elbow-worn sensors were stronger and more significant (p < 0.001) than those of the wrist-worn sensors. A new wireless assessment of surgical performance system for objective assessment of surgical skills has proven valid for assessing arthroscopic skills. The elbow-worn sensors were shown to achieve an accurate assessment of surgical dexterity and performance. The validation of an entirely objective assessment of arthroscopic skill with wireless elbow-worn motion sensors introduces, for the first time, a feasible assessment system for the live operating theater with the added potential to be applied to other surgical and interventional specialties.

A study of crowdsourced segment-level surgical skill assessment using pairwise rankings.

Currently available methods for surgical skills assessment are either subjective or only provide global evaluations for the overall task. Such global evaluations do not inform trainees about where in the task they need to perform better. In this study, we investigated the reliability and validity of a framework to generate objective skill assessments for segments within a task, and compared assessments from our framework using crowdsourced segment ratings from surgically untrained individuals and expert surgeons against manually assigned global rating scores. Our framework includes (1) a binary classifier trained to generate preferences for pairs of task segments (i.e., given a pair of segments, specification of which one was performed better), (2) computing segment-level percentile scores based on the preferences, and (3) predicting task-level scores using the segment-level scores. We conducted a crowdsourcing user study to obtain manual preferences for segments within a suturing and knot-tying task from a crowd of surgically untrained individuals and a group of experts. We analyzed the inter-rater reliability of preferences obtained from the crowd and experts, and investigated the validity of task-level scores obtained using our framework. In addition, we compared accuracy of the crowd and expert preference classifiers, as well as the segment- and task-level scores obtained from the classifiers. We observed moderate inter-rater reliability within the crowd (Fleiss' kappa, κ = 0.41) and experts (κ = 0.55). For both the crowd and experts, the accuracy of an automated classifier trained using all the task segments was above par as compared to the inter-rater agreement [crowd classifier 85 % (SE 2 %), expert classifier 89 % (SE 3 %)]. We predicted the overall global rating scores (GRS) for the task with a root-mean-squared error that was lower than one standard deviation of the ground-truth GRS. We observed a high correlation between segment-level scores (ρ ≥ 0.86) obtained using the crowd and expert preference classifiers. The task-level scores obtained using the crowd and expert preference classifier were also highly correlated with each other (ρ ≥ 0.84), and statistically equivalent within a margin of two points (for a score ranging from 6 to 30). Our analyses, however, did not demonstrate statistical significance in equivalence of accuracy between the crowd and expert classifiers within a 10 % margin. Our framework implemented using crowdsourced pairwise comparisons leads to valid objective surgical skill assessment for segments within a task, and for the task overall. Crowdsourcing yields reliable pairwise comparisons of skill for segments within a task with high efficiency. Our framework may be deployed within surgical training programs for objective, automated, and standardized evaluation of technical skills.

Development and validation of trauma surgical skills metrics: Preliminary assessment of performance after training.

Maintaining trauma-specific surgical skills is an ongoing challenge for surgical training programs. An objective assessment of surgical skills is needed. We hypothesized that a validated surgical performance assessment tool could detect differences following a training intervention. We developed surgical performance assessment metrics based on discussion with expert trauma surgeons, video review of 10 experts and 10 novice surgeons performing three vascular exposure procedures and lower extremity fasciotomy on cadavers, and validated the metrics with interrater reliability testing by five reviewers blinded to level of expertise and a consensus conference. We tested these performance metrics in 12 surgical residents (Year 3-7) before and 2 weeks after vascular exposure skills training in the Advanced Surgical Skills for Exposure in Trauma (ASSET) course. Performance was assessed in three areas as follows: knowledge (anatomic, management), procedure steps, and technical skills. Time to completion of procedures was recorded, and these metrics were combined into a single performance score, the Trauma Readiness Index (TRI). Wilcoxon matched-pairs signed-ranks test compared pretraining/posttraining effects. Mean time to complete procedures decreased by 4.3 minutes (from 13.4 minutes to 9.1 minutes). The performance component most improved by the 1-day skills training was procedure steps, completion of which increased by 21%. Technical skill scores improved by 12%. Overall knowledge improved by 3%, with 18% improvement in anatomic knowledge. TRI increased significantly from 50% to 64% with ASSET training. Interrater reliability of the surgical performance assessment metrics was validated with single intraclass correlation coefficient of 0.7 to 0.98. A trauma-relevant surgical performance assessment detected improvements in specific procedure steps and anatomic knowledge taught during a 1-day course, quantified by the TRI. ASSET training reduced time to complete vascular control by one third. Future applications include assessing specific skills in a larger surgeon cohort, assessing military surgical readiness, and quantifying skill degradation with time since training.

Objective Surgical Skill Assessment: An Initial Experience by Means of a Sensory Glove Paving the Way to Open Surgery Simulation?

Simulation and training in surgery are very promising tools for enhancing a surgeon's skill base. Accurate tracking of hand movements can be a strategy for objectively gauging a surgeon's dexterity, although "open" work is much more difficult to evaluate than are laparoscopic tasks. To the authors' knowledge, a system taking into account the movements of each finger joint has never been applied to open surgery simulation. This work intends to make up for this shortcoming and to perform a data analysis of the surgeon's entire gesture. The authors developed a sensory glove to measure flexion/extension of each finger joint and wrist movement. Totally 9 experts and 9 novices performed a basic suturing task and their manual performances were recorded within 2 days of measurements. Intraclass correlation coefficients were calculated to assess the ability of the executors to repeat and reproduce the proposed exercise. Wilcoxon signed-rank tests and Mann-Whitney U-tests were used to determine whether the 2 groups differ significantly in terms of execution time, repeatability, and reproducibility. Finally, a questionnaire was used to gather operators' subjective opinions. The experts needed a similar reduced execution time comparing the 2 recording sessions (p = 0.09), whereas novices spent more time during the first day (p = 0.01). Repeatability did not differ between the 2 days, either for experts (p = 0.26) or for novices (p = 0.86). The 2 groups performed differently in terms of time (p < 0.001), repeatability (p = 0.01), and reproducibility (p < 0.001) of the same gesture. The system showed an overall moderate repeatability (intraclass correlation coefficient: experts = 0.64; novices = 0.53) and an overall high reproducibility. The questionnaire revealed performers' positive feedback with the glove. This initial experience confirmed the validity and reliability of the proposed system in objectively assessing surgeons' technical skill, thus paving the way to a more complex project involving open surgery simulation.

Automated objective surgical skill assessment in the operating room from unstructured tool motion in septoplasty.

Previous work on surgical skill assessment using intraoperative tool motion has focused on highly structured surgical tasks such as cholecystectomy and used generic motion metrics such as time and number of movements. Other statistical methods such as hidden Markov models (HMM) and descriptive curve coding (DCC) have been successfully used to assess skill in structured activities on bench-top tasks. Methods to assess skill and provide effective feedback to trainees for unstructured surgical tasks in the operating room, such as tissue dissection in septoplasty, have yet to be developed. We proposed a method that provides a descriptive structure for septoplasty by automatically segmenting it into higher-level meaningful activities called strokes. These activities characterize the surgeon's tool motion pattern. We constructed a spatial graph from the sequence of strokes in each procedure and used its properties to train a classifier to distinguish between expert and novice surgeons. We compared the results from our method with those from HMM, DCC, and generic metric-based approaches. We showed that our method--with an average accuracy of 91 %--performs better or equal than these state-of-the-art methods, while simultaneously providing surgeons with an intuitive understanding of the procedure. In this study, we developed and evaluated an automated approach to objectively assess surgical skill during unstructured task of tissue dissection in nasal septoplasty.