Objective Assessment Of Skills Research Articles

Hysteroscopic Resection Training Utilizing a Virtual Reality and Low Fidelity Model: A Pilot Study

Faculty Advisor: Amy Boardman, MD PURPOSE: To evaluate effectiveness of Virtual Reality (VR) and low-fidelity inanimate model for training OB/GYN residents in hysteroscopic resection. BACKGROUND: While surgical training in gynecology is often accomplished through supervision in the operating room, simulation allows risk-free surgical skills practice. VR and low-fidelity models provide training of different surgical procedures, which could improve resident confidence and knowledge with less common procedures. As mechanical resection device use continues to increase, residents have fewer opportunities to train with hysteroscopic resectoscopes. METHODS: This prospective, cohort trial at Christiana Care, a large community hospital, enrolled 24 PGY1-4 OBGYN residents. All participants served as their own controls and underwent pre-intervention objective skills assessment testing (OSAT) for hysteroscopic resection using a low-fidelity potato model. Intervention included completion of a VR hysteroscopic simulator module to achieve a score >95%. Participants then underwent immediate post-intervention OSAT. All participants completed a subjective survey assessing comfort levels with the procedure pre- and post-intervention. All low-fidelity hysteroscopic resections were recorded and scored by a single, blinded evaluator. RESULTS: Seven OSAT surgical skills and subjective measures were assessed using a Likert Scale of 1 (low) to 5 (high). PGY 3&4 residents showed improvement in camera management, tissue handling, and instrument management (P=<.05). Three-month repeat OSATs showed retention of the same skills plus flow of procedure in all residents. Subjectively, there was significant improvement in comfort level of knowledge of procedural steps and energy, and using the loop resectoscope. DISCUSSION: Simulation improved baseline surgical skills and led to retention and improvement of same skills. Resident training should include simulation to increase both baseline development and retention of surgical skills.

Objective assessment of laparoscopic surgical skills

Objective assessment of laparoscopic surgical skills

Validation of a Simulation-training Model for Robotic Intracorporeal Bowel Anastomosis Using a Step-by-step Technique

ObjectiveTo develop and validate a training model for the robotic intracorporeal bowel anastomosis. MethodsFor simulation, surgeons with varying levels of experience were instructed about bowel anastomosis robotic surgical simulation in a short educational video. All participants performed the required steps for the intracorporeal bowel anastomosis under standardized conditions. The procedure consists of the following steps: division of the bowel with a stapler (1), incision and opening of the bowel limbs at the antimesenteric angle (2), insertion of the stapler into the 2 bowel limbs for the side-to-side anastomosis (3), and transverse closure of the anastomosis with the stapler (4). All simulations were performed using the daVinci SI robotic system. Face and content validity were assessed using a standardized questionnaire. Construct validity was evaluated using the Global Evaluative Assessment of Robotic Skills, a validated global performance rating scale. ResultsTwenty-two surgeons participated including 6 robotic experts and 16 trainees. The expert participants rated the bowel anastomosis model highly for face validity (median 4/5; 64% agree or strongly agree), and all participants rated the content as a training model very highly (median 4.5/5; 100% agree or strongly agree). Discrimination between experts and trainees using Global Evaluative Assessment of Robotic Skills demonstrated construct validity (novice 17.6 vs expert 24.7, P = .03). ConclusionWe demonstrate that the bowel anastomosis robotic surgical simulator is a reproducible and realistic simulation that allows for an objective skills assessment. We establish face, content, and construct validity for this model. This step-by-step technique may be utilized in training surgeons desiring to acquire skills in robotic intracorporeal urinary diversion.

SATR-DL: Improving Surgical Skill Assessment And Task Recognition In Robot-Assisted Surgery With Deep Neural Networks.

This paper focuses on an automated analysis of surgical motion profiles for objective skill assessment and task recognition in robot-assisted surgery. Existing techniques heavily rely on conventional statistic measures or shallow modelings based on hand-engineered features and gesture segmentation. Such developments require significant expert knowledge, are prone to errors, and are less efficient in online adaptive training systems. In this work, we present an efficient analytic framework with a parallel deep learning architecture, SATR-DL, to assess trainee expertise and recognize surgical training activity. Through an end-to-end learning technique, abstract information of spatial representations and temporal dynamics is jointly obtained directly from raw motion sequences. By leveraging a shared highlevel representation learning, the resulting model is successful in the recognition of trainee skills and surgical tasks, suturing, needle-passing, and knot-tying. Meanwhile, we explore the use of ensemble in classification at the trial level, where the SATR-DL outperforms state-of-the-art performance by achieving accuracies of 0.960 and 1.000 in skill assessment and task recognition, respectively. This study highlights the potential of SATR-DL to provide improvements for an efficient data-driven assessment in intelligent robotic surgery.

High Fidelity Virtual Reality Orthognathic Surgery Simulator.

Surgical simulators are powerful tools that assist in providing advanced training for complex craniofacial surgical procedures and objective skills assessment such as the ones needed to perform Bilateral Sagittal Split Osteotomy (BSSO). One of the crucial steps in simulating BSSO is accurately cutting the mandible in a specific area of the jaw, where surgeons rely on high fidelity visual and haptic cues. In this paper, we present methods to simulate drilling and cutting of the bone using the burr and the motorized oscillating saw respectively. Our method allows low computational cost bone drilling or cutting while providing high fidelity haptic feedback that is suitable for real-time virtual surgery simulation.

Teaching Vaginal Hysterectomy via Simulation: Creation and Validation of the Objective Skills Assessment Tool for Simulated Vaginal Hysterectomy on a Task Trainer and Performance Among Different Levels of Trainees.

The use of simulation-based education to train surgeons is widely accepted. Although many authors describe the use of an Objective Skills Assessment Test (OSAT) to assess performance of various surgical procedures, there is a paucity of research on use of this modality to evaluate vaginal surgery skills. We created a vaginal hysterectomy procedure-specific checklist (PSC) to complete the OSAT (which is composed of a PSC and a global rating scale [GRS]). The primary objective of this study is to evaluate the performance of a novel evaluation strategy for vaginal hysterectomy using an OSAT combining PSC and GRS. This is a descriptive prospective study from a single institution. After orientation to the model, participants were filmed performing vaginal hysterectomy. A blinded grader scored each subject using the PSC and GRS. Medical students, residents, fellows, and attendings performed vaginal hysterectomy on a simulated model. Mean PSC and GRS scores increased significantly with surgeon level of experience (P < 0.001). Procedure-specific checklist scores significantly correlated with GRS scores (P < 0.001). The vaginal hysterectomy model and PSC have been studied across different surgeon levels using OSATs. Training programs should consider using this low-cost task trainer as a teaching tool.

Contribution to the objective assessment of technical skills for surgery students: An accelerometer based approach

Aim of studyGestural skills assessment in odontology is a highly complex task. Although mandatory for certification, proficiency assessment is still required for parameters that are more objective. Our aim was to assess whether accelerometer sensors might contribute to distinguishing efficiency in therapeutic gesture from experts and novices in the realization of surgical sutures. Material and method3 groups of participants: novices (n = 8), intermediaries (n = 14) and experts (n = 7) were enrolled in the study. They had to perform different types of odontological suture. We used 2 wireless wrist sensors to measure objectively the acceleration of both hands and to deduce the economy and the fluidity of the movement. A video recording was also used to complete the assessment. ResultsThe time and motion criteria were documented to significantly correlate with the experience of the participant. The total time factor has a significant effect with the experience level (p = 0.006). The fluidity of the movement has also been shown to vary significantly between groups. ConclusionThe use of accelerometers coupled with image analysis could make it possible to envisage in the long run an objective evaluation for this type of surgical gesture.

Impact of Escalating Cognitive Workload and Temporal Demands on Surgeons Cognitive Function

Impact of Escalating Cognitive Workload and Temporal Demands on Surgeons Cognitive Function

Teaching in the operating room: trends in surgical skills transfer in ophthalmology

Cataract surgery is arguably the most commonly performed operation in ophthalmology. Surgical skills transfer from experienced surgeons to resident surgeons is complicated by the fact that the teaching surgeon primarily acts as an observer rather than directly performing the procedure. Therefore, wet lab and simulator training are utilized to reduce the learning curve of the novice surgeons, which establishes tissue awareness, dexterity and muscle memory required to perform each step of the procedure, safely. Access to a wet lab and simulator environment is accomplished by establishing a surgical training curriculum in residency programs. In the operating room, topical anesthesia is a safe alternative for teaching cataract surgery. There are three well-described approaches to teaching individual steps of cataract surgery: forward, “backwards”, and deconstructed step-by-step instruction. Simulator training can be incorporated prior to live patient experience or integrated concurrently with learner presence in the operating room. The trend towards a competency-based instruction model has necessitated appropriate evaluation tools that include Objective Assessment of Skills in Intraocular Surgery (OASIS), Global Rating Assessment of Skills in Intraocular Surgery (GRASIS), and the International Council of Ophthalmology’s Ophthalmology Surgical Competency Assessment Rubrics (ICO-OSCAR). We review the literature on trends in surgical teaching in ophthalmology, with the focus on cataract surgery instruction to the novice surgeon.

Objective Assessment of Orthopaedic Skills

International Journal of Medicine and Public Health,2017,7,4,187-190.DOI:10.5530/ijmedph.2017.4.39Published:November 2017Type:Original ArticleObjective Assessment of Orthopaedic SkillsSimerjit Singh, Avneet Kaur, and Harmanpreet Singh Simerjit Singh1*, Avneet Kaur2, Harmanpreet Singh3 1MS, Orthopaedics, Associate Professor, Department of Orthopaedics, Universiti Tunku Abdul Rahman, Kuala Lumpur, MALAYSIA. 2Lecturer, Department of Physiology, Universiti Tunku Abdul Rahman, Kuala Lumpur, MALAYSIA. 3Associate Professor, Department of Pharmacy, Lovely Professional University, Jalandhar, INDIA. Abstract:Introduction: We have been using task trainers for teaching various techniques such as injections for carpal tunnel, trigger finger and tenosynovitis and knee lavage. However, at present these skills are being taught in a nonsystematic manner and we do not have an objective method of evaluating them. The aim of this study was to device an objective tool for measurement of these skills. Methods: A total of 80 fourth year MBBS students participated in the study. A MCQ test was used for base line assessment of student’s knowledge of the knee anatomy. The students were divided into three groups for instruction on knee joint injection i.e. didactic (lecture), video and hands on training group. The three groups performed a task of simulated knee joint injection on an anatomical knee model and assessed using a newly prepared assessment tool. A second MCQ test was conducted after the joint injection task. Each participant was asked to rate his or her confidence in performing knee joint injections with use of l0-point Likert scale before and after the instruction. Results: On comparing the means by one-way Anova, the mean score was highest for the hands on training group (26.32) followed by video group (20.46) and lecture group (16.15) respectively. The difference in the scores between three groups was statistically significant (p<0.000). Participants in all three groups showed gain in knowledge and confidence at performing the task. The overall internal consistency (cronbach’s alpha) of the assessment tool developed was 0.63. However high convergent validity (inter-item correlation) was found for 8 out of 9 items of the assessment tool. Conclusion: Availability of an objective assessment tool for orthopaedic skills will systemize the delivery and evaluation of these skills. Further the experience gained through this study will be used to develop objective assessment tools to evaluate the skills in other medical disciplines. Keywords:Injection skills., Joint injection, Knee arthrocentesis, Knee injection, Procedural skills, Skill assessmentView:PDF (134.06 KB)

E-learning teaches attendings “how to” objectively assess pediatric urology trainees' surgery skills for orchiopexy

Established methods to train pediatric urology surgery by residency training programs require updating in response to administrative changes such as new, reduced trainee duty hours. Therefore, new objective methods must be developed to teach trainees. We approached this need by creating e-learning to teach attendings objective assessment of trainee skills using the Zwisch scale, an established assessment tool. The aim of this study was to identify whether or not e-learning is an appropriate platform for effective teaching of this assessment tool, by assessing inter-rater correlation of assessments made by the attendings after participation in the e-learning. Pediatric orchiopexy was used as the index case. An e-learning tool was created to teach attending surgeons objective assessment of trainees' surgical skills. First, e-learning content was created which showed the assessment method videotape of resident surgery done in the operating room. Next, attendings were enrolled to e-learn this method. Finally, the ability of enrollees to assess resident surgery skill performance was tested. Namely, test video was made showing a trainee performing inguinal orchiopexy. All enrollees viewed the same online videos. Assessments of surgical skills (Zwisch scale) were entered into an online survey. Data were analyzed by intercorrelation coefficient kappa analysis (strong correlation was ICC≥0.7). A total of 11 attendings were enrolled. All accessed the online learning and then made assessments of surgical skills trainees showed on videotapes. The e-learning comprised three modules: 1. "Core concepts," in which users learned the assessment tool methods; 2. "Learn to assess," in which users learned how to assess by watching video clips, explaining the assessment method; and 3. "Test," in which users tested their skill at making assessments by watching video clips and then actively inputting their ratings of surgical and global skills as viewed in the video clips (Figure). A total of 89 surgical skill ratings were performed with 56 (65%) exact matches between raters and 89 (100%) matched within one rank. Interclass correlation coefficient (ANOVA) showed statistically significant correlation. (r=0.725, 95% CI 0.571-0.837, F=3.976, p≤0.00001). Kappa analysis of inter-rater reliability showed strong consensus between attendings for average measures with ICC=0.71, 95% CI 0.46-0.95 (p=0.03). We launched e-learning to teach pediatric urology attendings "how to" assess trainee surgical skills objectively (Zwisch scale). After e-learning, there was strong inter-rater correlation in assessments made. We plan to extend such e-learning to pediatric urology surgical training programs.

Objective assessment of colonoscope manipulation skills in colonoscopy training.

Manipulation of the colonoscope is a technical challenge for novice clinicians which is best learned in a simulated environment. It involves the coordination of scope tip steering with scope insertion, using a rotated image as reference. The purpose of this work is to develop and validate a system which objectively assesses colonoscopy technical skills proficiency in an arbitrary training environment, allowing novices to assess their technical proficiency prior to real patient encounters. We implemented a motion tracking setup to objectively analyze and assess the way operators perform colonoscopies, including an analysis of wrist and elbow joint motions. Subsequently, we conducted a validation study to verify whether our motion analysis could discriminate novice colonoscopists from experts. Participants navigated a wooden bench-top model using a standard colonoscope while their motions were tracked. The developed motion tracking setup allowed colonoscopists of varying levels of proficiency to have their colonoscope manipulation assessed, and was able to be operated by a trained non-technical operator. Novice operators had significantly greater median times (101.5 vs. 31.5s) and number of hand movements (62.0 vs. 21.5) than experts. Experts, however, spent a significantly greater proportion of time in extreme ranges of wrist and elbow joint motion than novices. We have developed and implemented a hand and joint motion analysis system that is able to discriminate novices from experts based on objective measures of motion. These metrics could, thus, serve as proxies for technical proficiency during training.

Effectiveness of Immersive Virtual Reality in Surgical Training—A Randomized Control Trial

Surgical training methods are evolving with the technological advancements, including the application of virtual reality (VR) and augmented reality. However, 28 to 40% of novice residents are not confident in performing a major surgical procedure. VR surgery, an immersive VR (iVR) experience, was developed using Oculus Rift and Leap Motion devices (Leap Motion, Inc, San Francisco, CA) to address this challenge. Our iVR is a multisensory, holistic surgical training application that demonstrates a maxillofacial surgical technique, the Le Fort I osteotomy. The main objective of the present study was to evaluate the effect of using VR surgery on the self-confidence and knowledge of surgical residents. A multisite, single-blind, parallel, randomized controlled trial (RCT) was performed. The participants were novice surgical residents with limited experience in performing the Le Fort I osteotomy. The primary outcome measures were the self-assessment scores of trainee confidence using a Likert scale and an objective assessment of the cognitive skills. Ninety-five residents from 7 dental schools were included in the RCT. The participants were randomly divided into a study group of 51 residents and a control group of 44. Participants in the study group used the VR surgery application on an Oculus Rift with Leap Motion device. The control group participants used similar content in a standard PowerPoint presentation on a laptop. Repeated measures multivariate analysis of variance was applied to the data to assess the overall effect of the intervention on the confidence of the residents. The study group participants showed significantly greater perceived self-confidence levels compared with those in the control group (P=.034; α=0.05). Novices in the first year of their training showed the greatest improvement in their confidence compared with those in their second and third year. iVR experiences improve the knowledge and self-confidence of the surgical residents.

Objective Assessment of Technical Skills in Otorhinolaryngology-Head and Neck Surgery Residents: A Systematic Review.

Objectives The primary goal is the indexation of validated methods used to assess surgical competency in otorhinolaryngology-head and neck surgery (ORL-HNS) residents. Secondary goals include assessment of the reliability and validity of these tools, as well as the documentation of specific procedures in ORL-HNS involved. Data Sources MEDBASE, OVID, Medline, CINAHL, and EBM, as well as the printed references, available through the Université de Montréal library. Review Methods The PRISMA method was used to review digital and printed databases. Publications were reviewed by 2 independent reviewers, and selected articles were fully analyzed to classify evaluation methods and categorize them by procedure and subspecialty of ORL-HNS involved. Reliability and validity were assessed and scored for each assessment tool. Results Through the review of 30 studies, 5 evaluation methods were described and validated to assess the surgical competency of ORL-HNS residents. The evaluation method most often described was the combined Global Rating Scale and Task-Specific Checklist tool. Reliability and validity for this tool were overall high; however, considerable data were unavailable. Eleven distinctive surgical procedures were studied, encompassing many subspecialties of ORL-HNS: facial plastics, general ear-nose-throat, laryngology, otology, pediatrics, and rhinology. Conclusions Although assessment tools have been developed for an array of surgical procedures, involving most ORL-HNS subspecialties, the use of combined checklists has been repeatedly validated in the literature and shown to be easily applicable in practice. It has been applied to many ORL-HNS procedures but not in oncologic surgery to date.

Activity Recognition for Quality Assessment of Batting Shots in Cricket using a Hierarchical Representation

Quality assessment in cricket is a complex task that is performed by understanding the combination of individual activities a player is able to perform and by assessing how well these activities are performed. We present a framework for inexpensive and accessible, automated recognition of cricketing shots. By means of body-worn inertial measurement units, movements of batsmen are recorded, which are then analysed using a parallelised, hierarchical recognition system that automatically classifies relevant categories of shots as required for assessing batting quality. Our system then generates meaningful visualisations of key performance parameters, including feet positions, attack/defence, and distribution of shots around the ground. These visualisations are the basis for objective skill assessment thereby focusing on specific personal improvement points as identified through our system. We evaluated our framework through a deployment study where 6 players engaged in batting exercises. Based on the recorded movement data we could automatically identify 20 classes of unique batting shot components with an average F1-score greater than 88%. This analysis is the basis for our detailed analysis of our study participants’ skills. Our system has the potential to rival expensive vision-based systems but at a fraction of the cost.

Evaluation of Skill Assessment After Hysteroscopy Training Curriculum a designed

Background: To evaluate a standard latex hysteroscopy model training course on gynecologists’ surgical skills. Methods: 74 gynecologists who attended to hysteroscopy training courses of Training Centre, Tehran University, 2013 to 2015 were investigated. Theoretical information of trainees were obtained at the beginning, at the end and 6 months after the course by filling up a reliabile and valid questionnaire. and its subjective and objective impact on their surgical skills and satisfaction were evaluated. Results: 87.84% of participants were satisfied by the curriculum and for most of them, virtual uteruses were felt realistic. After 6 months, subject impact of curriculum was 77.03%. But, 60.81% of gynecologists hardly could perform hysteroscopy, which was mostly due to lack of technology (53.22%)(Harders, 2006 #11). Objective assessment of surgical skills was done for those who could perform hysteroscopy. Of them 58.7% had experienced a significant increase in their surgical skills (p=0.004). Correlation of subjective and objective impact of curriculum was low (r=0.43), (p=0.003). Conclusion: Hysteroscopy training program can improve gynecologists’ hysteroscopic skills. The only index which is significantly correlated with the subjective impact of these courses is the number of previous operations. Also in Iran, because of lacking hysteroscopy equipment in most hospitals, the effect of training courses decreases after a while. So we recommend to train hysteroscopy in all medical universities and equip all hospitals with hysteroscope. Also we suggest training courses for all gynecologists and repeted it every some years.

Modern Metrics for Evaluating Surgical Technical Skills

Assessment of surgical technical skills has traditionally occurred through informal means using sporadic and subjective feedback. Recent trends in patient safety, simulation training, and technological advances have generated a need for accurate and objective assessment of surgical technical skills. This review is intended to identify the breadth and scope of recent approaches to surgical skill assessment and metrics and define gaps for further development. Surgical skill assessment methods can be broadly categorized as (1) assessments by expert observations using validated task-specific checklists and global rating scales, (2) automated motion tracking metrics using mechanical force sensors, accelerometers or computer video analysis, and (3) mechanical outcome metrics. Surgical skill assessments are likely to play an ever-increasing role in surgical training and certification. Further research is needed to determine the optimal metrics for assessment and to identify relevant real-world outcomes.

Automated robot-assisted surgical skill evaluation: Predictive analytics approach.

Surgical skill assessment has predominantly been a subjective task. Recently, technological advances such as robot-assisted surgery have created great opportunities for objective surgical evaluation. In this paper, we introduce a predictive framework for objective skill assessment based on movement trajectory data. Our aim is to build a classification framework to automatically evaluate the performance of surgeons with different levels of expertise. Eight global movement features are extracted from movement trajectory data captured by a da Vinci robot for surgeons with two levels of expertise - novice and expert. Three classification methods - k-nearest neighbours, logistic regression and support vector machines - are applied. The result shows that the proposed framework can classify surgeons' expertise as novice or expert with an accuracy of 82.3% for knot tying and 89.9% for a suturing task. This study demonstrates and evaluates the ability of machine learning methods to automatically classify expert and novice surgeons using global movement features.

Robotic Pancreatoduodenectomy Biotissue Curriculum has Validity and Improves Technical Performance for Surgical Oncology Fellows

Obtaining the proficiency on the robotic platform necessary to safely perform a robotic pancreatoduodenectomy is particularly challenging. We hypothesize that by instituting a proficiency-based robotic training curriculum we can enhance novice surgeons' skills outside of the operating room, leading to a shorter learning curve. A biotissue curriculum was designed consisting of sewing artificial organs to simulate a hepaticojejunostomy (HJ), gastrojejunostomy (GJ), and pancreaticojejunostomy (PJ). Three master robotic surgeons performed each biotissue anastomosis to assess validity. Using video review, trainee performance on biotissue drills was evaluated for time, errors and objective structured assessment of technical skills (OSATS) by 2 blinded graders. This study is conducted at the University of Pittsburgh Medical Center (Pittsburgh, PA), a tertiary care academic teaching hospital. In total, 14 surgical oncology fellows completed the biotissue curriculum. Fourteen fellows performed 196 anastomotic drills during the first year: 66 (HJ), 64 (GJ), and 66 (PJ). The fellows' performances were analyzed as a group by attempt. The attendings' first attempt outperformed the fellows' first attempt in all metrics for every drill (all p < 0.05). More than 5 analyzed attempts of the HJ, there was improvement in time, errors, and OSATS (all p < 0.01); however, no metric reached attending performance. For the GJ, time, errors, and OSATS all improved more than 5 attempts (all p < 0.01), whereas only errors and OSATS reached proficiency. For the PJ, errors and OSATS both improved over attempts (p < 0.01) and reached proficiency; however, time did not statistically improve nor reach proficiency. The graders scoring correlated for errors and OSATS (p < 0.0001). A pancreatoduodenectomy biotissue curriculum has face and construct validity. The curriculum is feasible and improves errors and technical performance. Time is the most difficult technical parameter to improve. This curriculum is a valid tool for teaching robotic pancreatoduodenectomies with established milestones for reaching optimum performance.

How does practice improve the skills of medical students during consecutive training courses?

To follow-up the development of medical students in taking a stitch using two different bench models, and to assess their performance with regards to gender, handedness, prior recreational activities and interest in surgery. The study was performed during the compulsory basic (n=152) and the consecutive elective course (n=27). Students took simple interrupted stitches into synthetic and biopreprate models in the classroom and in the operating room. The time needed for that was measured and the quality was scored, using an OSATS (Objective Structured Assessment of Technical Skill) checklist that had been adapted to our training programs. Students' performance improved both in time and quality during classes, over the course and compared basic to the elective course, too. No significant difference was found in relation to gender and handedness but certain recreational activities and special interest in surgery led to better results. Operating room environment had a slight negative effect on students' performance. The study could provide objective skill assessment, monitoring has revealed deficiencies and influencing factors. Objective feedback, valid and reliable assessment is important in teaching surgical skills. In addition it may contribute to higher surgical safety later on in the clinical practice.