Evaluation Of Surgical Skills Research Articles

Automated assessment of non-technical skills by heart-rate data.

Observer-based scoring systems, or automatic methods, based on features or kinematic data analysis, are used to perform surgical skill assessments. These methods have several limitations, observer-based ones are subjective, and the automatic ones mainly focus on technical skills or use data strongly related to technical skills to assess non-technical skills. In this study, we are exploring the use of heart-rate data, a non-technical-related data, to predict values of an observer-based scoring system thanks to random forest regressors. Heart-rate data from 35 junior resident orthopedic surgeons were collected during the evaluation of a meniscectomy performed on a bench-top simulator. Each participant has been evaluated by two assessors using the Arthroscopic Surgical Skill Evaluation Tool (ASSET) score. A preprocessing stage on heart-rate data, composed of threshold filtering and a detrending method, was considered before extracting 41 features. Then a random forest regressor has been optimized thanks to a randomized search cross-validation strategy to predict each score component. The prediction of the partially non-technical-related components presents promising results, with the best result obtained for the safety component with a mean absolute error of 0.24, which represents a mean absolute percentage error of 5.76%. The analysis of feature important allowed us to determine which features are the more related to each ASSET component, and therefore determine the underlying impact of the sympathetic and parasympathetic nervous systems. In this preliminary work, a random forest regressor train on feature extract from heart-rate data could be used for automatic skill assessment and more especially for the partially non-technical-related components. Combined with more traditional data, such as kinematic data, it could help to perform accurate automatic skill assessment.

Reply to: Letter to the Editor on: Hypospadias Reconstruction Training: Development of an Ex-Vivo Model and Objective Evaluation of Surgical Skills

Reply to: Letter to the Editor on: Hypospadias Reconstruction Training: Development of an Ex-Vivo Model and Objective Evaluation of Surgical Skills

Letter to the Editor on "Hypospadias Reconstruction Training: Development of an Ex-Vivo Model and Objective Evaluation of Surgical Skills"

Letter to the Editor on "Hypospadias Reconstruction Training: Development of an Ex-Vivo Model and Objective Evaluation of Surgical Skills"

Editorial Comment on "Hypospadias Reconstruction Training: Development of an Ex-Vivo Model and Objective Evaluation of Surgical Skills"

Editorial Comment on "Hypospadias Reconstruction Training: Development of an Ex-Vivo Model and Objective Evaluation of Surgical Skills"

Reply to Editorial Comment on “Hypospadias Reconstruction Training: Development of an Ex-Vivo Model and Objective Evaluation of Surgical Skills”

Reply to Editorial Comment on “Hypospadias Reconstruction Training: Development of an Ex-Vivo Model and Objective Evaluation of Surgical Skills”

Objective performance indicators versus GEARS: an opportunity for more accurate assessment of surgical skill.

Surgical skill evaluation that relies on subjective scoring of surgical videos can be time-consuming and inconsistent across raters. We demonstrate differentiated opportunities for objective evaluation to improve surgeon training and performance. Subjective evaluation was performed using the Global evaluative assessment of robotic skills (GEARS) from both expert and crowd raters; whereas, objective evaluation used objective performance indicators (OPIs) derived from da Vinci surgical systems. Classifiers were trained for each evaluation method to distinguish between surgical expertise levels. This study includes one clinical task from a case series of robotic-assisted sleeve gastrectomy procedures performed by a single surgeon, and two training tasks performed by novice and expert surgeons, i.e., surgeons with no experience in robotic-assisted surgery (RAS) and those with more than 500 RAS procedures. When comparing expert and novice skill levels, OPI-based classifier showed significantly higher accuracy than GEARS-based classifier on the more complex dissection task (OPI 0.93 ± 0.08 vs. GEARS 0.67 ± 0.18; 95% CI, 0.16-0.37; p = 0.02), but no significant difference was shown on the simpler suturing task. For the single-surgeon case series, both classifiers performed well when differentiating between early and late group cases with smaller group sizes and larger intervals between groups (OPI 0.9 ± 0.08; GEARS 0.87 ± 0.12; 95% CI, 0.02-0.04; p = 0.67). When increasing the group size to include more cases, thereby having smaller intervals between groups, OPIs demonstrated significantly higher accuracy (OPI 0.97 ± 0.06; GEARS 0.76 ± 0.07; 95% CI, 0.12-0.28; p = 0.004) in differentiating between the early/late cases. Objective methods for skill evaluation in RAS outperform subjective methods when (1) differentiating expertise in a technically challenging training task, and (2) identifying more granular differences along early versus late phases of a surgeon learning curve within a clinical task. Objective methods offer an opportunity for more accessible and scalable skill evaluation in RAS.

Neuromechanisms of simulation-based arthroscopic skills assessment: a fNIRS study.

The neural mechanisms underlying differences in the performance of simulated arthroscopic skills across various skill levels remain unclear. Our primary objective is to investigate the learning mechanisms of simulated arthroscopic skills using functional near-infrared spectroscopy (fNIRS). We recruited 27 participants, divided into three groups: novices (n = 9), intermediates (n = 9), and experts (n = 9). Participants completed seven arthroscopic tasks on a simulator, including diagnostic navigation, triangulation, grasping stars, diagnostic exploration, meniscectomy, synovial membrane cleaning, and loose body removal. All tasks were videotaped and assessed via the simulator system and the Arthroscopic Surgical Skill Evaluation Tool (ASSET), while cortical activation data were collected using fNIRS. Simulator scores and ASSET scores were analyzed to identify different level of performance of all participants. Brain region activation and functional connectivity (FC) of different types of participants were analyzed from fNIRS data. Both the expert and intermediate groups scored significantly higher than the novice group (p < 0.001). There were significant differences in ASSET scores between experts and intermediates, experts and novices, and intermediates and novices (p = 0.0047, p < 0.0001, p < 0.0001), with the trend being experts > intermediates > novices. The intermediate group exhibited significantly greater activation in the left primary motor cortex (LPMC) and left prefrontal cortex (LPFC) compared to the novice group (p = 0.0152, p = 0.0021). Compared to experts, the intermediate group demonstrated significantly increased FC between the presupplementary motor area (preSMA) and the right prefrontal cortex (RPFC; p < 0.001). Additionally, the intermediate group showed significantly increased FC between the preSMA and LPFC, RPFC and LPFC, and LPMC and LPFC compared to novices (p = 0.0077, p = 0.0285, p = 0.0446). Cortical activation and functional connectivity reveal varying levels of activation intensity in the PFC, PMC, and preSMA among novices, intermediates, and experts. The intermediate group exhibited the highest activation intensity.

Hypospadias Reconstruction Training: Development of an Ex-Vivo Model for Objective Evaluation of Surgical Skills

To objectively evaluate technical skill acquisition in hypospadias repair procedures during surgical training using noninvasive wearable sensor technology. We combined subjective video evaluations with objective electromyography (EMG) measurements in a hands-on hypospadias training course. Surgeons wore wireless EMG and accelerometer sensors on their dominant hand while performing tasks on ex-vivo cadaveric calf penises. The study focused on 4 skills as follows: urethral mobilization, dorsal inlay graft harvest/implantation, meatal-based flap urethroplasty, and dorsal plication. Machine learning techniques analyzed muscle activation patterns and attributes for assessing surgical precision. The course included 18 participants (10 female, 8 males; average age 40.18±8.46 years) categorized as novice (n=10, <3 years' experience), intermediate (n=5, 3-5 years), and expert (n=3, >5 years). Video evaluations did not reveal significant differences due to short-term training. However, EMG measurements showed significant reductions in average EMG power, total time, dominant frequency, and cumulative muscle workload after training. Additionally, the mean power spectral density of the EMG signal decreased notably post-training. This study presents a structured approach for hypospadias training and highlights the effectiveness of wearable sensor technology for objective skill assessment. While video evaluations did not detect significant changes, EMG data provided measurable differences in skill acquisition, suggesting that wearable sensors could enhance objective evaluations of surgical proficiency in residency programs.

Classification of subtask types and skill levels in robot-assisted surgery using EEG, eye-tracking, and machine learning

BackgroundObjective and standardized evaluation of surgical skills in robot-assisted surgery (RAS) holds critical importance for both surgical education and patient safety. This study introduces machine learning (ML) techniques using features derived from electroencephalogram (EEG) and eye-tracking data to identify surgical subtasks and classify skill levels.MethodThe efficacy of this approach was assessed using a comprehensive dataset encompassing nine distinct classes, each representing a unique combination of three surgical subtasks executed by surgeons while performing operations on pigs. Four ML models, logistic regression, random forest, gradient boosting, and extreme gradient boosting (XGB) were used for multi-class classification. To develop the models, 20% of data samples were randomly allocated to a test set, with the remaining 80% used for training and validation. Hyperparameters were optimized through grid search, using fivefold stratified cross-validation repeated five times. Model reliability was ensured by performing train-test split over 30 iterations, with average measurements reported.ResultsThe findings revealed that the proposed approach outperformed existing methods for classifying RAS subtasks and skills; the XGB and random forest models yielded high accuracy rates (88.49% and 88.56%, respectively) that were not significantly different (two-sample t-test; P-value = 0.9).ConclusionThese results underscore the potential of ML models to augment the objectivity and precision of RAS subtask and skill evaluation. Future research should consider exploring ways to optimize these models, particularly focusing on the classes identified as challenging in this study. Ultimately, this study marks a significant step towards a more refined, objective, and standardized approach to RAS training and competency assessment.

Can a Low-Fidelity Arthroscopic Simulator Improve Technical Expertise in Performing Temporomandibular Joint Arthroscopy?

BackgroundChallenges in temporomandibular joint (TMJ) arthroscopy training include the cost of operating room time, surgical risks, and the accessibility of cadavers and high-fidelity simulators. A low-fidelity simulator (LFS) was developed for initial TMJ arthroscopy training. PurposeThe aim of this study was to evaluate improvement in TMJ arthroscopic skills after training with an LFS using the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale. Study design, setting, sampleA prospective randomized controlled study was conducted. Subjects included Oral and Maxillofacial Surgery residents in post-graduate year (PGY) 1–5 at Thomas Jefferson University. Chief residents actively performing TMJ arthroscopy were excluded. Predictor variableThe predictor variable was participation in LFS training. Subjects were randomly assigned to the LFS training or no LFS training group. Main outcome variablesThe primary outcome variable was simulated surgical skill measured by total ASSET score (maximum score of 35). The secondary outcome variables were simulated surgical skill measured by each ASSET domain score and time to completion (TTC). CovariatesThe covariates included sex, age, handedness, PGY, endoscopic experience, open TMJ surgery experience, and experience with musical instruments and sports. AnalysesData analyses included paired T-tests to determine changes in outcome variables after TMJ arthroscopy training between experimental groups, and ANOVA and chi-square tests to identify associations between covariates. Inter-rater reliability of the blinded examiners was evaluated using Cronbach’s alpha correlation. A P-value < .05 was considered significant. ResultsThe sample was composed of 10 residents: 5 (100%) males in the no LFS group, 3 (60%) males and 2 (40%) females in the LFS group (P = .4). There was no significant difference between the groups in pre-TMJ arthroscopy training ASSET scores (P = .3). After training, the total ASSET score improved by 3.40 ± 6.87 in the no LFS group and by 6.27 ± 2.68 in the LFS group (P = .03). Conclusions and relevanceStudy results showed that low-fidelity simulation can improve fundamental arthroscopic skills. Future research should be conducted on a larger scale to further validate this model and show the transfer of skill improvements to cadaver and live patient settings.

A surgical instrument motion measurement system for skill evaluation in practical laparoscopic surgery training.

This study developed and validated a surgical instrument motion measurement system for skill evaluation during practical laparoscopic surgery training. Owing to the various advantages of laparoscopic surgery including minimal invasiveness, this technique has been widely used. However, expert surgeons have insufficient time for providing training to beginners due to the shortage of surgeons and limited working hours. Skill transfer efficiency has to be improved for which there is an urgent need to develop objective surgical skill evaluation methods. Therefore, a simple motion capture-based surgical instrument motion measurement system that could be easily installed in an operating room for skill assessment during practical surgical training was developed. The tip positions and orientations of the instruments were calculated based on the marker positions attached to the root of the instrument. Because the patterns of these markers are individual, this system can track multiple instruments simultaneously and detect exchanges. However due to the many obstacles in the operating room, the measurement data included noise and outliers. In this study, the effect of this decrease in measurement accuracy on feature calculation was determined. Accuracy verification experiments were conducted during wet-lab training to demonstrate the capability of this system to measure the motion of surgical instruments with practical accuracy. A surgical training experiment on a cadaver was conducted, and the motions of six surgical instruments were measured in 36 cases of laparoscopic radical nephrectomy. Outlier removal and smoothing methods were also developed and applied to remove the noise and outliers in the obtained data. The questionnaire survey conducted during the experiment confirmed that the measurement system did not interfere with the surgical operation. Thus, the proposed system was capable of making reliable measurements with minimal impact on surgery. The system will facilitate surgical education by enabling the evaluation of skill transfer of surgical skills.

Development of the "LAser Surgical skills Evaluation for Residents" (LASER) Scale Through a Modified Delphi Method.

The CO2 laser is frequently used during microlaryngeal surgery (MLS) for a variety of pathology including laryngeal malignancy and stenosis. Learning how to use the laser safely is part of the curriculum for every otolaryngology resident. However, assessment of laryngoscopy technical skills can be challenging for supervisors, making it difficult to adequately provide feedback to trainees. "LAser Surgical skills Evaluation for Residents" (LASER) Scale aims to facilitate the evaluation of residents' performance and promote constructive feedback. The initial evaluation grid was based on a literature review of CO2 laser laryngoscopy (with an emphasis on indications, technique, safety, and efficacy) using Covidence systematic review software (Veritas Health Innovation). The final version was produced after 4 rounds of Delphi surveys. This study was an international collaboration including 15 otolaryngologists with either laryngology or head and neck surgery subspecialties. Panelists were based in Canada (8), the United States (3), France (1), Spain (1), Belgium (1), and Lebanon (1). The process involved 4 rounds of Delphi surveys. Assessment categories included: anesthesia considerations, pre- and perioperative laser safety measures, and surgical technique. Consensus was reached on final survey completion. Through a modified Delphi method, a novel scale was developed through an international collaborative effort that evaluates resident skillset in CO2 laser MLS. Future studies are warranted to validate this assessment tool.

Developing a Training Model for Local Flaps Using Fresh Human Skin Excised During Tummy Tuck Procedures

Introduction: The proficient execution of local flaps poses a challenge for surgical residents. This study aimed to assess a training model for local flaps using fresh human skin excised during tummy tuck procedures. Methods: A questionnaire and a surgical skills evaluation, examining theoretical and procedural knowledge of local flaps, were administered before and after the training. All surgical procedures were performed on a simulation model utilizing freshly excised human skin. Skills were evaluated using a modified version of the Objective Structured Assessment of Technical Skills Score. SPSS version 21 was employed for comparing pre- and post-training results. Results: In the pretraining assessment, residents faced difficulties in flap design accuracy and wound coverage adequacy, indicating a need for surgical training beyond the operating theater. Post-training, there was a significant improvement in procedural skills, as reflected in the modified Objective Structured Assessment of Technical Skills score. The mean cumulative pretraining score was 26.81 ± 5.41, while the posttraining score was 43.59 ± 5.72 (P = 0.008). Theoretical knowledge also significantly improved in the posttraining evaluation, except for the indication of a Z-plasty (P = 0.257). The training model received positive feedback and was deemed highly useful and recommendable. Discussion: The presented model demonstrated improved surgical handling and understanding of tissue rotation under realistic conditions. The simulation model, using fresh human skin, is cost-effective and versatile for a broad range of flap procedures, advocating for its further adoption. Conclusions: The training model significantly enhanced surgical handling and comprehension of tissue rotation, replicating realistic conditions. The cost-effective simulation model, based on fresh human skin, offers versatility for various flap procedures, emphasizing the need for its wider implementation.

Reliability in performance assessment creates a potential application of artificial intelligence in veterinary education: evaluation of suturing skills at a single institution.

To evaluate suturing skills of veterinary students using 3 common performance assessments (PAs) and to compare findings to data obtained by an electromyographic armband. 16 second-year veterinary students. Students performed 4 suturing tasks on synthetic tissue models 1, 3, and 5 weeks after a surgical skills course. Digital videos were scored by 4 expert surgeons using 3 PAs (an Objective Structured Clinical Examination [OSCE]- style surgical binary checklist, an Objective Structured Assessment of Technical Skill [OSATS] checklist, and a surgical Global Rating Scale [GRS]). Surface electromyography (sEMG) data collected from the dominant forearm were input to machine learning algorithms. Performance assessment scores were compared between experts and correlated to task completion times and sEMG data. Inter-rater reliability was calculated using the intraclass correlation coefficient (ICC). Inter-rater agreement was calculated using percent agreement with varying levels of tolerance. Reliability was moderate for the OSCE and OSATS checklists and poor for the GRS. Agreement was achieved for the checklists when moderate tolerance was applied but remained poor for the GRS. sEMG signals did not correlate well with checklist scores or task times, but features extracted from signals permitted task differentiation by routine statistical comparison and correct task classification using machine learning algorithms. Reliability and agreement of an OSCE-style checklist, OSATS checklist, and surgical GRS assessment were insufficient to characterize suturing skills of veterinary students. To avoid subjectivity associated with PA by raters, further study of kinematics and EMG data is warranted in the surgical skills evaluation of veterinary students.

Current update on multimodal analgesia and nonopiate surgical pain management

Based on the Competency Assessment Tool, herein we developed an assessment instrument suitable to evaluate the implantation of central intravenous devices.Surgical assessment is based mainly on the subjective impressions of the teacher. Based on the “Competency Assessment Tool” (CAT) developed for the evaluation of technical surgical skills in minimally invasive colorectal resection, we designed an assessment tool suitable to evaluate the implantation of central venous access devices performed by junior surgical trainees.Four major assessments during the different steps of the intervention were used in this evaluation. Each of these tasks was divided into four sub-domains according to surgical skill. In addition to the CAT score, the apprentices’ skills were evaluated using a visual assessment that was quantified using an analogue scale (value from 1 to 10). The candidates were classified into junior and senior trainees depending on the number of procedures they had already performed and on their surgical experience.71 procedures were evaluated during the study period. Seven senior trainees conducted 43 procedures and five junior trainees performed 28 interventions. The senior trainees had significantly higher CAT scores than junior candidates, and the scores fluctuated according to surgical experience, usually reaching their peak after 10 procedures.The CAT model is well suited for the assessment of surgical trainees during central venous access device implantation. It enables a close assessment of the learning process and the technical skills of trainees, which helps them improving in a safe, standardized manner.

Evaluation of single-stage vision models for pose estimation of surgical instruments.

Multiple applications in open surgical environments may benefit from adoption of markerless computer vision depending on associated speed and accuracy requirements. The current work evaluates vision models for 6-degree of freedom pose estimation of surgical instruments in RGB scenes. Potential use cases are discussed based on observed performance. Convolutional neural nets were developed with simulated training data for 6-degree of freedom pose estimation of a representative surgical instrument in RGB scenes. Trained models were evaluated with simulated and real-world scenes. Real-world scenes were produced by using a robotic manipulator to procedurally generate a wide range of object poses. CNNs trained in simulation transferred to real-world evaluation scenes with a mild decrease in pose accuracy. Model performance was sensitive to input image resolution and orientation prediction format. The model with highest accuracy demonstrated mean in-plane translation error of 13mm and mean long axis orientation error of 5[Formula: see text] in simulated evaluation scenes. Similar errors of 29mm and 8[Formula: see text] were observed in real-world scenes. 6-DoF pose estimators can predict object pose in RGB scenes with real-time inference speed. Observed pose accuracy suggests that applications such as coarse-grained guidance, surgical skill evaluation, or instrument tracking for tray optimization may benefit from markerless pose estimation.

Effectiveness of the Eyesi Surgical Simulator for ophthalmology trainees: systematic review and meta-analysis

ObjectifLe simulateur chirurgical Eyesi (VRmagic, Mannheim, Allemagne) est le simulateur de réalité virtuelle le plus souvent utilisé en chirurgie ophtalmologique. Si des études ont révélé que des outils de formation fondés sur la simulation améliorent l'acquisition des compétences, aucune revue de synthèse systématique avec méta-analyse n'a encore été réalisée à l’égard du simulateur chirurgical Eyesi. Notre étude visait donc à effectuer une méta-analyse des études qui comparaient le simulateur chirurgical Eyesi à d'autres méthodes d'enseignement des compétences techniques auprès d’étudiants. MéthodesNotre recherche a porté sur 3 bases de données (MEDLINE, Embase et CENTRAL). Les articles inclus comparaient le simulateur chirurgical Eyesi à d'autres formes d'enseignement des compétences techniques auprès d’étudiants et de résidents dans des facultés de médecine. On a réalisé une méta-analyse par paire qui reposait sur l'inverse de la variance à effets aléatoires. Les différences moyennes standardisées (DMS) ont servi de principal paramètre de mesure afin de tenir compte des différences entre les outils d’évaluation des compétences chirurgicales. RésultatsDes 207 citations évaluées, on a retenu 8 études, totalisant 108 étudiants qui ont eu recours au simulateur chirurgical Eyesi et de 123 étudiants qui ont utilisé une autre forme d'apprentissage. L'amélioration des scores de compétences techniques était significativement supérieure dans le groupe Eyesi (3 études; DMS = 2,02; intervalle de confiance [IC] à 95 % : 1,47–2,57; p < 0,001; I2 = 0 %). On a noté une baisse significative du nombre d’étudiants du groupe Eyesi qui ont fait des erreurs techniques après leur formation (5 études : rapport de cotes = 0,43; IC à 95 % : 0,20–0,90; p = 0,03; I2 = 30 %). Aucune différence significative n'a été enregistrée entre les groupes en ce qui a trait au temps consacré pour exécuter une tâche après la formation (3 études; DMS = 1,96; IC à 95 % : –1,96–5,88; p = 0,33; I2 = 97 %). ConclusionsLe simulateur chirurgical Eyesi pourrait améliorer l'acquisition des compétences techniques (faible degré de certitude des données probantes) et réduire le nombre d'erreurs techniques (très faible degré de certitude des données probantes) lors de l'apprentissage de la chirurgie de la cataracte chez des étudiants. Il y a donc lieu de réaliser d'autres études prospectives sur le module Eyesi qui comporteraient un système de notation standardisée.

Craniotomy Simulator with Force Myography and Machine Learning-Based Skills Assessment.

Craniotomy is a fundamental component of neurosurgery that involves the removal of the skull bone flap. Simulation-based training of craniotomy is an efficient method to develop competent skills outside the operating room. Traditionally, an expert surgeon evaluates the surgical skills using rating scales, but this method is subjective, time-consuming, and tedious. Accordingly, the objective of the present study was to develop an anatomically accurate craniotomy simulator with realistic haptic feedback and objective evaluation of surgical skills. A CT scan segmentation-based craniotomy simulator with two bone flaps for drilling task was developed using 3D printed bone matrix material. Force myography (FMG) and machine learning were used to automatically evaluate the surgical skills. Twenty-two neurosurgeons participated in this study, including novices (n = 8), intermediates (n = 8), and experts (n = 6), and they performed the defined drilling experiments. They provided feedback on the effectiveness of the simulator using a Likert scale questionnaire on a scale ranging from 1 to 10. The data acquired from the FMG band was used to classify the surgical expertise into novice, intermediate and expert categories. The study employed naïve Bayes, linear discriminant (LDA), support vector machine (SVM), and decision tree (DT) classifiers with leave one out cross-validation. The neurosurgeons' feedback indicates that the developed simulator was found to be an effective tool to hone drilling skills. In addition, the bone matrix material provided good value in terms of haptic feedback (average score 7.1). For FMG-data-based skills evaluation, we achieved maximum accuracy using the naïve Bayes classifier (90.0 ± 14.8%). DT had a classification accuracy of 86.22 ± 20.8%, LDA had an accuracy of 81.9 ± 23.6%, and SVM had an accuracy of 76.7 ± 32.9%. The findings of this study indicate that materials with comparable biomechanical properties to those of real tissues are more effective for surgical simulation. In addition, force myography and machine learning provide objective and automated assessment of surgical drilling skills.

MP26-09 EYE MOVEMENT BEHAVIOR ASSOCIATES WITH EXPERTISE LEVEL IN ROBOT-ASSISTED SURGERY

You have accessJournal of UrologyCME1 Apr 2023MP26-09 EYE MOVEMENT BEHAVIOR ASSOCIATES WITH EXPERTISE LEVEL IN ROBOT-ASSISTED SURGERY Mehdi Seilanian Toussi, Saeed Shadpour, Kristopher Attwood, Qian Liu, Camille Gutierrez, james L. Mohler, and Somayeh B. Shafiei Mehdi Seilanian ToussiMehdi Seilanian Toussi More articles by this author , Saeed ShadpourSaeed Shadpour More articles by this author , Kristopher AttwoodKristopher Attwood More articles by this author , Qian LiuQian Liu More articles by this author , Camille GutierrezCamille Gutierrez More articles by this author , james L. Mohlerjames L. Mohler More articles by this author , and Somayeh B. ShafieiSomayeh B. Shafiei More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003254.09AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: An objective method for assessing robot-assisted surgery (RAS) skills that can be used in clinical settings has not been established. This study aimed to compare visual metrics at three different skill levels—inexperienced, competent, or experienced— to determine whether eye gaze data differed among RAS surgeons. METHODS: Eleven right-handed participants who performed coagulation subtasks (monopolar or bipolar cautery) during hysterectomy, cystectomy, or nephrectomy on live pigs with the Da Vinci robot had their eye gaze data captured using TobiiPro2 eyeglasses. Visual metrics were extracted using eye gaze data (Table 1). An expert RAS surgeon evaluated each participant's performance and level of expertise after watching operation videos using the modified Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. Each metric was compared across the three skill levels—novice, competent, or experienced—using ANOVA. RESULTS: When participants performed coagulation using the dominant hand, the entropy of the right eye’s pupil diameter differed among skill levels (p-value = 0.01). When this subtask was completed with the non-dominant hand, the rates of change of the gaze directions of the left eye in the horizontal and vertical dimensions and the right eye in the vertical dimension differed (p-values = 2x10-6, 6x10-6, and 6x10-4, respectively) across skill levels. Table 2 shows the significance level for the difference in visual metrics between skill level pairs. CONCLUSIONS: Visual metrics differed significantly across RAS skill levels. The ease and efficacy of recording and evaluating visual data without interfering with the surgeon's work suggest a new objective surgical skill evaluation method that could be applied to clinical settings. Source of Funding: National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award number R01EB029398, the Alliance Foundation of Roswell Park Comprehensive Cancer Center, and the National Cancer Institute (NCI) grant P30CA016056 involving the use of Roswell Park Comprehensive Cancer Center’s Laboratory Animal Shared Resource and Applied Technology Laboratory for Advanced Surgery © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e355 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Mehdi Seilanian Toussi More articles by this author Saeed Shadpour More articles by this author Kristopher Attwood More articles by this author Qian Liu More articles by this author Camille Gutierrez More articles by this author james L. Mohler More articles by this author Somayeh B. Shafiei More articles by this author Expand All Advertisement PDF downloadLoading ...

Deep neural network architecture for automated soft surgical skills evaluation using objective structured assessment of technical skills criteria.

Classic methods of surgery skills evaluation tend to classify the surgeon performance in multi-categorical discrete classes. If this classification scheme has proven to be effective, it does not provide in-between evaluation levels. If these intermediate scoring levels were available, they would provide more accurate evaluation of the surgeon trainee. We propose a novel approach to assess surgery skills on a continuous scale ranging from 1 to 5. We show that the proposed approach is flexible enough to be used either for scores of global performance or several sub-scores based on a surgical criteria set called Objective Structured Assessment of Technical Skills (OSATS). We established a combined CNN+BiLSTM architecture to take advantage of both temporal and spatial features of kinematic data. Our experimental validation relies on real-world data obtained from JIGSAWS database. The surgeons are evaluated on three tasks: Knot-Tying, Needle-Passing and Suturing. The proposed framework of neural networks takes as inputs a sequence of 76 kinematic variables and produces an output float score ranging from 1 to 5, reflecting the quality of the performed surgical task. Our proposed model achieves high-quality OSATS scores predictions with means of Spearman correlation coefficients between the predicted outputs and the ground-truth outputs of 0.82, 0.60 and 0.65 for Knot-Tying, Needle-Passing and Suturing, respectively. To our knowledge, we are the first to achieve this regression performance using the OSATS criteria and the JIGSAWS kinematic data. An effective deep learning tool was created for the purpose of surgical skills assessment. It was shown that our method could be a promising surgical skills evaluation tool for surgical training programs.