Objective Assessment Of Technical Skills Research Articles

Enhancing proficiency in ascites puncture through procedural simulation: an audit type study investigating medium-term skill retention.

Medical simulation has become an essential teaching method for all health professionals. It not only allows to acquire technical and non-technical knowledge, but also helps the maintenance of acquired knowledge in the medium and long term. Ascites puncture is part of the basic technical procedures learned by medical students during their internship. To evaluate the role of simulation-based learning of ascites puncture on the improvement of theoretical knowledge and maintenance of skills at 3 months. We conducted an audit type study with two cycles of data collection at the simulation center at the Faculty of Medicine of Sousse between November 2020 and June 2021. We included learners in their third year of medical studies who had a hospital internship in the gastroenterology department at Sahloul Hospital in Sousse. All learners attended the initial simulation session on ascites fluid puncture. Thereafter, they were free to accept or refuse participation in the evaluation session that was scheduled after 3 months, depending on their availability. Forty learners participated in the procedural simulation of the ascites fluid puncture technique. Thirty-four (85%) were female and six (5%) were male. In our study, we showed that following procedural simulation training of ascites puncture, there was a significant improvement in the theoretical knowledge of the learners (p < 0.000). Objective assessment of technical skills after 3 months showed the benefit of performance maintenance (p < 0.000). Our study confirmed the benefit of simulation-based learning on the improvement of theoretical knowledge and the maintenance of technical performance in the medium term.

MP68-06 MOTION CAPTURE SYSTEM FOR OBJECTIVE ASSESSMENT OF TECHNICAL SKILLS DURING URETEROSCOPY

You have accessJournal of UrologyCME1 Apr 2023MP68-06 MOTION CAPTURE SYSTEM FOR OBJECTIVE ASSESSMENT OF TECHNICAL SKILLS DURING URETEROSCOPY Brittany Berk, Mahdi Ebnali-Heidari, Christian Miccile, Roger Dias, and Daniel Wollin Brittany BerkBrittany Berk More articles by this author , Mahdi Ebnali-HeidariMahdi Ebnali-Heidari More articles by this author , Christian MiccileChristian Miccile More articles by this author , Roger DiasRoger Dias More articles by this author , and Daniel WollinDaniel Wollin More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003331.06AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Prior studies have demonstrated success with the use of motion analysis technology to assess surgical psychomotor skills. This study aims to provide preliminary evidence on the feasibility of using motion capture (mocap) system for objective assessment of a surgeon’s technical skills during ureteroscopic procedures. METHODS: Motion data of two participants, an attending urologist and a novice with no prior surgical training, were collected using the Rokoko smart suit with a sampling rate of 100 frames per second. This mocap system consists of 19 sensors; 9 Degree of Freedom (DoF) inertial measurement units each connected via wires to a hub for wireless communication (Figure 1). After data collection and preprocessing, data were analyzed to extract kinematic information from multiple joints including the wrist, elbow, and shoulder during a simulated ureteroscopic procedure. RESULTS: These data show that the distribution of x, y, and z locations of specific body part sensors can be visualized to investigate the variables associated with movement including speed, acceleration, the economy of movement, and smoothness. We found a distinct pattern in joint kinematics such as wrist, elbow flexion, extension, and pronation of the expert when compared to our novice participant. Interestingly, for most metrics, the expert showed a larger range of motion compared to the novice (Figure 1). To further investigate how these fluctuations are reflected in the frequency domains, we used Power Spectral Density (PSD) analysis, displaying the distribution of power of motion signals across distinct frequency bands. We found a higher power of motion in the signal associated with the expert left wrist flexion motion, indicating more energy spent on this motion pattern compared to the novice participant. CONCLUSIONS: In this study, we demonstrate that a mocap system is a feasible method for understanding surgeon motion patterns during ureteroscopic procedures. With additional participants and more complex simulated surgical tasks, we expect to utilize this framework to better delineate the ways that urologic surgeons utilize endoscopic equipment and to define metrics for the assessment of efficient kinematics related to surgical quality. This technology may also prove useful in urological surgical training in the future. Source of Funding: None © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e954 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Brittany Berk More articles by this author Mahdi Ebnali-Heidari More articles by this author Christian Miccile More articles by this author Roger Dias More articles by this author Daniel Wollin More articles by this author Expand All Advertisement PDF downloadLoading ...

Force-based assessment of tissue handling skills in simulation training for robot-assisted surgery

IntroductionAlthough robotic-assisted surgery is increasingly performed, objective assessment of technical skills is lacking. The aim of this study is to provide validity evidence for objective assessment of technical skills for robotic-assisted surgery.MethodsAn international multicenter study was conducted with participants from the academic hospitals Heidelberg University Hospital (Germany, Heidelberg) and the Amsterdam University Medical Centers (The Netherlands, Amsterdam). Trainees with distinctly different levels of robotic surgery experience were divided into three groups (novice, intermediate, expert) and enrolled in a training curriculum. Each trainee performed six trials of a standardized suturing task using the da Vinci Surgical System. Using the ForceSense system, five force-based parameters were analyzed, for objective assessment of tissue handling skills. Mann–Whitney U test and linear regression were used to analyze performance differences and the Wilcoxon signed-rank test to analyze skills progression.ResultsA total of 360 trials, performed by 60 participants, were analyzed. Significant differences between the novices, intermediates and experts were observed regarding the total completion time (41 s vs 29 s vs 22 s p = 0.003), mean non zero force (29 N vs 33 N vs 19 N p = 0.032), maximum impulse (40 Ns vs 31 Ns vs 20 Ns p = 0.001) and force volume (38 N3 vs 32 N3 vs 22 N3p = 0.018). Furthermore, the experts showed better results in mean non-zero force (22 N vs 13 N p = 0.015), maximum impulse (24 Ns vs 17 Ns p = 0.043) and force volume (25 N3 vs 16 N3p = 0.025) compared to the intermediates (p ≤ 0.05). Lastly, learning curve improvement was observed for the total task completion time, mean non-zero force, maximum impulse and force volume (p ≤ 0.05).ConclusionConstruct validity for force-based assessment of tissue handling skills in robot-assisted surgery is established. It is advised to incorporate objective assessment and feedback in robot-assisted surgery training programs to determine technical proficiency and, potentially, to prevent tissue trauma.

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill.

This paper introduces a new dataset of a surgical knot-tying task, and a multi-modal deep learning model that achieves comparable performance to expert human raters on this skill assessment task. Seventy-two surgical trainees and faculty were recruited for the knot-tying task, and were recorded using video, kinematic, and image data. Three expert human raters conducted the skills assessment using the Objective Structured Assessment of Technical Skill (OSATS) Global Rating Scale (GRS). We also designed and developed three deep learning models: a ResNet-based image model, a ResNet-LSTM kinematic model, and a multi-modal model leveraging the image and time-series kinematic data. All three models demonstrate performance comparable to the expert human raters on most GRS domains. The multi-modal model demonstrates the best overall performance, as measured using the mean squared error (MSE) and intraclass correlation coefficient (ICC). This work is significant since it demonstrates that multi-modal deep learning has the potential to replicate human raters on a challenging human-performed knot-tying task. The study demonstrates an algorithm with state-of-the-art performance in surgical skill assessment. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.

The use of objective assessments in the evaluation of technical skills in cardiothoracic surgery: a systematic review.

Summary OBJECTIVESWith reductions in training time and intraoperative exposure, there is a need for objective assessments to measure trainee progression. This systematic review focuses on the evaluation of trainee technical skill performance using objective assessments in cardiothoracic surgery and its incorporation into training curricula.METHODSDatabases (EBSCOHOST, Scopus and Web of Science) and reference lists of relevant articles for studies that incorporated objective assessment of technical skills of trainees/residents in cardiothoracic surgery were included. Data extraction included task performed; assessment setting and tool used; number/level of assessors; study outcome and whether the assessments were incorporated into training curricula. The methodological rigour of the studies was scored using the Medical Education Research Study Quality Instrument (MERSQI).RESULTSFifty-four studies were included for quantitative synthesis. Six were randomized-controlled trials. Cardiac surgery was the most common speciality utilizing objective assessment methods with coronary anastomosis the most frequently tested task. Likert-based assessment tools were most commonly used (61%). Eighty-five per cent of studies were simulation-based with the rest being intraoperative. Expert surgeons were primarily used for objective assessments (78%) with 46% using blinding. Thirty (56%) studies explored objective changes in technical performance with 97% demonstrating improvement. The other studies were primarily validating assessment tools. Thirty-nine per cent of studies had established these assessment tools into training curricula. The mean ± standard deviation MERSQI score for all studies was 13.6 ± 1.5 demonstrating high validity.CONCLUSIONSDespite validated technical skill assessment tools being available and demonstrating trainee improvement, their regular adoption into training curricula is lacking. There is a need to incorporate these assessments to increase the efficiency and transparency of training programmes for cardiothoracic surgeons.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents’ Technical Skills

The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor. The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed. The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles. The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial. ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Dimensionless squared jerk: An objective differential to assess experienced and novice probe movement in obstetric ultrasound.

ObjectiveWidely accepted, validated and objective measures of ultrasound competency have not been established for clinical practice. Outcomes of training curricula are often based on arbitrary thresholds, such as the number of clinical cases completed. We aimed to define metrics against which competency could be measured.MethodWe undertook a prospective, observational study of obstetric sonographers at a UK University Teaching Hospital. Participants were either experienced in fetal ultrasound (n = 10, >200 ultrasound examinations) or novice operators (n = 10, <25 ultrasound examinations). We recorded probe motion data during the performance of biometry on a commercially available mid‐trimester phantom.ResultsWe report that Dimensionless squared jerk, an assessment of deliberate hand movements, independent of movement duration, extent, spurious peaks and dimension differed significantly different between groups, 19.26 (SD 3.02) for experienced and 22.08 (SD 1.05, p = 0.01) for novice operators, respectively. Experienced operator performance, was associated with a shorter time to task completion of 176.46 s (SD 47.31) compared to 666.94 s (SD 490.36, p = 0.0004) for novice operators. Probe travel was also shorter for experienced operators 521.23 mm (SD 27.41) versus 2234.82 mm (SD 188.50, p = 0.007) when compared to novice operators.ConclusionOur results represent progress toward an objective assessment of technical skill in obstetric ultrasound. Repeating this methodology in a clinical environment may develop insight into the generalisability of these findings into ultrasound education.

Objective assessment of technical skill targeted to time in cataract surgery.

To evaluate time as an objective measure of technical skill assessment in cataract surgery. Single academic center. Cross-sectional study. One hundred videos of cataract surgery (29 performed by a faculty and 71 performed by trainee surgeons) performed at the Wilmer Eye Institute between July 2011 and December 2017 were analyzed for surgical technical skill as a function of time using task-specific indices, global skill indices, and appointment status. Expert surgeons, defined by global skill, did not consistently have shorter times to complete phases in cataract surgery compared with novice surgeons. Time to complete phases in cataract surgery was predictive of global skill (area under the curve [AUC] of 0.73, 95% CI: 0.63-0.83) and of surgeons' appointment status (AUC 0.82, 95% CI: 0.70-0.91). Time for capsulorhexis correctly distinguished between expert-novice skill class only 58% (AUC 0.58, 95% CI: 0.47-0.69) of the times when capsulorhexis indices were used and 51% (AUC 0.51, 95% CI: 0.39-0.63) of the times when global indices were used. Time to complete quadrants in capsulorhexis also had a low ability to discriminate between novice and expert instances for capsulorhexis skill and global skill. The AUC was 0.54 (95% CI: 0.44-0.66) and 0.53 (95% CI: 0.41-0.65) for capsulorhexis skill and global skill, respectively. Time as an objective measure of skill was limited in its ability to distinguish between expert and novice skill class in a meaningful way.

Contribution to the objective assessment of technical skills for surgery students: An accelerometer based approach

Aim of studyGestural skills assessment in odontology is a highly complex task. Although mandatory for certification, proficiency assessment is still required for parameters that are more objective. Our aim was to assess whether accelerometer sensors might contribute to distinguishing efficiency in therapeutic gesture from experts and novices in the realization of surgical sutures. Material and method3 groups of participants: novices (n = 8), intermediaries (n = 14) and experts (n = 7) were enrolled in the study. They had to perform different types of odontological suture. We used 2 wireless wrist sensors to measure objectively the acceleration of both hands and to deduce the economy and the fluidity of the movement. A video recording was also used to complete the assessment. ResultsThe time and motion criteria were documented to significantly correlate with the experience of the participant. The total time factor has a significant effect with the experience level (p = 0.006). The fluidity of the movement has also been shown to vary significantly between groups. ConclusionThe use of accelerometers coupled with image analysis could make it possible to envisage in the long run an objective evaluation for this type of surgical gesture.

Objective Assessment of Technical Skills in Otorhinolaryngology-Head and Neck Surgery Residents: A Systematic Review.

Objectives The primary goal is the indexation of validated methods used to assess surgical competency in otorhinolaryngology-head and neck surgery (ORL-HNS) residents. Secondary goals include assessment of the reliability and validity of these tools, as well as the documentation of specific procedures in ORL-HNS involved. Data Sources MEDBASE, OVID, Medline, CINAHL, and EBM, as well as the printed references, available through the Université de Montréal library. Review Methods The PRISMA method was used to review digital and printed databases. Publications were reviewed by 2 independent reviewers, and selected articles were fully analyzed to classify evaluation methods and categorize them by procedure and subspecialty of ORL-HNS involved. Reliability and validity were assessed and scored for each assessment tool. Results Through the review of 30 studies, 5 evaluation methods were described and validated to assess the surgical competency of ORL-HNS residents. The evaluation method most often described was the combined Global Rating Scale and Task-Specific Checklist tool. Reliability and validity for this tool were overall high; however, considerable data were unavailable. Eleven distinctive surgical procedures were studied, encompassing many subspecialties of ORL-HNS: facial plastics, general ear-nose-throat, laryngology, otology, pediatrics, and rhinology. Conclusions Although assessment tools have been developed for an array of surgical procedures, involving most ORL-HNS subspecialties, the use of combined checklists has been repeatedly validated in the literature and shown to be easily applicable in practice. It has been applied to many ORL-HNS procedures but not in oncologic surgery to date.

Assessment of central venous catheterization in a simulated model using a motion-tracking device: an experimental validation study.

BackgroundCentral venous catheterization (CVC) is a basic requirement for many medical specialties. Simulated training in CVC may allow the acquisition of this competency but few reports have established a valid methodology for learning and acquiring procedural skills for CVC. This study aims to validate the use of a tracking motion device, the imperial college surgical assessment device (ICSAD), by comparing it with validated global rating scales (GRS) to measure CVC performance in a simulated torso.MethodsSenior year medical students, first and last year residents (PGY1, LYR), and expert anesthesiologists performed a jugular CVC assessment in a simulated model (Laerdal IV Torso). A validated GRS for objective assessment of technical skills and motion analysis by ICSAD was used. Statistical analysis was performed through Mann–Whitney and Kruskal–Wallis tests for construct validity and Spearman correlation coefficients between the ICSAD and GRS scores for concurrent validity between both.Results32 subjects were recruited (10 medical students, 8 PGY1, 8 LYR and 8 experts). Total path length measured with ICSAD and GRS scores were significantly different between all groups, except for LYR compared to experts (p = 0.664 for GRS and p = 0.72 for ICSAD). Regarding jugular CVC procedural time, LYR and experts were faster than PGY1 and MS (p < 0.05). Spearman correlation coefficient was −0.684 (p < 0.001) between ICSAD and GRS scores.ConclusionsICSAD is a valid tool for assessment of jugular CVC since it differentiates between expert and novice subjects, and correlates with a validated GRS for jugular CVC in a simulated torso.

Simulation for Medical Specialty Initial Certification and Maintenance of Certification in the United States of America

Surgical skills simulation (SSS) tests the application of factual knowledge and shows how knowledge is applied, representing the second and third levels of Miller’s Pyramid of Learning. SSS permits high-stakes scenario testing in safe environments. Therefore, SSS incorporation into initial specialty certification began in 2002 in Australia and New Zealand. The United States began SSS incorporation into specialty certification in 2008. This paper will determine where the United States stands in the process of SSS incorporation into specialty certification. Google scholar Internet and PubMed searches phrased “medical board certification surgical skills simulation”, performed on September 1, 2016 yielded 16 relevant articles. Hand search on September 1, 2016 yielded 7 additional articles. In 2008, cardiac catheterization simulation was required for interventional cardiology maintenance of certification (MOC). In 2010 the American Board of Anesthesiology (ABA) required SSS as part of the MOC program. In 2014, the summative assessment, Colorectal Objective Assessment of Technical Skills became part of the American Board of Colon and Rectal Surgery certification. In 2017, SSS will be added to the ABA initial certification examination. The United States has been slow to incorporate SSS into initial certification and MOC. Assessment validation, capital and recurring costs, personnel, physical facility and time requirements are barriers limiting SSS expansion into specialty certification processes. As SSS allows rapid technical skill assessment, without posing a threat to patients, expansion of SSS into initial certification and MOC programs represents non-maleficence and beneficence, and should be encouraged.

Validation of an oncoplastic breast simulator for assessment of technical skills in wide local excision.

Simulation enables safe practice and facilitates objective assessment of technical skills. However, simulation training in breast surgery is rare and assessment remains subjective. The primary aim was to evaluate the construct validity of technical skills assessments in wide local excision (WLE). Surgeons of different grades performed a WLE of a 25-mm palpable tumour on an in-house synthetic breast simulator. Procedures were videotaped (blinded), reviewed retrospectively, and independently rated against a procedure-specific global rating scale by two consultant breast surgeons. Specimen radiographs were obtained and the macroscopic distance from the 'tumour' edge to the resection margin was recorded in four cardinal directions. Expert consensus was used to construct an Oncoplastic Deviation Score (ODS), assigning points for excessively wide (more than 10 mm) and, conversely, close (less than 5 mm) macroscopic margins. Thirty-four surgeons (12 consultant surgeons, 12 specialty trainees and 10 core trainees) participated in the study. Video-based rating scores varied hierarchically with operator expertise (P < 0.050). Inter-rater reliability was excellent (α ≥ 0.80, P < 0.050 for all scales), and inter-rater agreement was moderate (κ = 0.132-0.361, P < 0.050 for all scales). Statistically significant differences were observed on pairwise comparisons between each grade of surgeon in scores for 'exposure', 'skin flap development', 'glandular remodelling', 'skin closure' and 'final product review' (P < 0.050). Consultants received significantly fewer ODS points than specialty trainees (P = 0.012) and core trainees (P = 0.028). Compared with experts (median 9.0 mm), wider margins were observed amongst specialty trainees (median 12.0 mm) and narrower margins amongst core trainees (median 7.1 mm) (P = 0.001). Video ratings of performance and a proposed ODS differentiate surgeons based on technical skills in WLE and may be useful for objective assessment of breast surgery trainees.

21. Assessing technical skills in oncoplastic breast surgery: Procedure specific global rating scales for wide local excision are construct valid

21. Assessing technical skills in oncoplastic breast surgery: Procedure specific global rating scales for wide local excision are construct valid

Design of a Box Trainer for Objective Assessment of Technical Skills in Single-port Surgery

Laparoscopic single-port (SP) surgery uses only a single entry point for all instruments. The approach of SP has been applied in multiple laparoscopic disciplines owing to its improved cosmetic result. However, in SP surgery, instrument movements are further restricted, resulting in increased instrument collisions compared with standard multiport (MP) laparoscopy. Our goal was to develop a trainer that can quantitatively measure task time, force and motion data during both MP and SP training to investigate the influence of instrument configuration on performance. Custom-made abdominal force sensors and accelerometers were integrated into a new training box that can be used in an SP and an MP configuration. This new box trainer measures forces, acceleration, and tilt angles during training of SP and MP laparoscopy. With the new trainer, 13 novices performed a tissue manipulation task to test whether significant differences exist between MP and SP in maximum abdominal force, maximum tissue manipulation force, maximum acceleration, and tilt angles of the handles. The results show that the task time (SP-145s, standard deviation (SD) = 103 vs MP-61s SD = 16), maximum abdominal force (SP-8.4N, SD = 2.0 vs MP-left (L)-3.3N, SD = 0.8 and MP-right (R)-5.8N, SD = 2.1), tissue manipulation force (SP-10.4N, SD = 3.6 and MP-5.6N, SD = 1.3), maximum acceleration (MP-L-9m/s(2), SD = 5 vs SP-L-14m/s(2), SD = 7), and tilt angles of the left handle are significantly higher in SP. This study shows that the new trainer can be used to find the most important differences in instrument and tissue handling, which is an important step toward the assessment of surgical skills needed for safe SP surgery depending on force and motion-based parameters.

Introduction of a Structured Assessment of Clinical Competency for Fellows in Gynecologic Oncology: A Pilot Study

Background: Gynecologic oncology surgery is recognized as a highly complex surgical specialty. Technical skill is the most critical expertise that a gynecology oncologist must acquire. Objective assessment of these skills is both valuable and necessary. Objectives: This observational cohort study study had two objectives: (1) to demonstrate the feasibility of running national assessments for technical and communication skills for gynecologic oncology fellows (GOFs) in an annual conference setting; and (2) to demonstrate the design of a technical-skills assessment examination relevant to GOFs. Materials and Methods: All fellows in attendance at the conference were invited to participate in the Objective Assessment of Technical Skills (OSATS) study. Expert examiners evaluated each skills station. Results: Eight, of a possible 14, volunteer fellows participated in the pilot test. There was a statistical difference between candidates for laparoscopic vault closure (p=0.029). The global rating scale was able to determine a statistically significant difference skill level between Year 1 and Year 2 fellows (p=0.016). Laparoscopic suturing and breaking bad news were the competencies identified as requiring most improvement for fellows. Conclusions: It is feasible to assess GOFs' technical and communication skills objectively as part of a national continuing education meeting. Devoting further resources to objective skills evaluation is justified for GOFs. (J GYNECOL SURG 31:17)

Abstract P2-18-18: Improving training in breast surgical oncology: The development and preliminary validation of a simulator for wide local excision

Abstract Background Simulation training has the potential to increase resident confidence in basic procedural tasks prior to operating room (OR) refinements, offsets a reduction in mentoring opportunities, avoids the use of patients for skills practice, and facilitates objective assessment of technical skills. Despite a multitude of benefits, adoption of simulation training and assessment in breast surgical oncology has not been forthcoming. Aims To develop and validate a high-fidelity surgical simulator for the practice, training and assessment of technical skills in wide local excision (WLE) of a palpable breast tumour. Methods Subjects (attendings and residents with an interest in breast surgery, and novice trainees) were invited to perform a WLE of 25mm palpable breast lesion located 30mm from the nipple areolar complex in the 3o clock position, on a synthetic breast simulator developed at Imperial College London as part of the London Deanery Skills Programme. Procedures were videotaped (blind) and were retrospectively reviewed and independently rated against procedure-specific ratings of performance (VAS 0-100) by two expert breast surgeons (&amp;gt;10 years experience). Specimen radiographs were performed (BioVision, Faxitron, USA) and margin of clearance (mm) were calculated from the edge of the “tumour” to the limits of surrounding breast tissue excised in 4 cardinal directions (i.e. N, S, E W). Specimen weights were recorded (g). Subjects completed a comprehensive questionnaire to determine simulator content and face validity. Results 21 subjects participated (5 attendings, 13 registrars, and 3 junior trainees). Data was analysed according to experience (high = &amp;gt;100 independent wide local procedures; low = no independent wide local procedures). Statistically significant (p ≤ 0.05) differences in performance were observed between high and low experience surgeons [for each category data are VAS score medians±IQR; ‘exposure’: low = 70.0±50.0, high = 80.0±20.0; ‘skin flap development’: low = 70.0±50.0, high = 80.0±40.0; ‘resection skills’: low = 60.0±55.0, high = 80.0±45.0; ‘glandular remodelling’: low = 70.0±25.0; high = 80.0±50.0, ‘skin closure’: low = 70.0±25.0, high = 90.0±20.0]. Resection margin width and specimen weight (MWU = 52.5, p = 1.0) did not discriminate high-low experience surgeons. The majority of participants believed that training on the model simulated the same steps as a real WLE (94.7%) and was useful for real practice (84.2%). Discussion Preliminary data on a WLE simulator suggests that the model is face and construct valid, and may be useful to supplement early stage practice prior to skills development in the OR. Video performance ratings whilst time consuming appear to distinguish high-low experience surgeons better than do specimen weight or radiographic margin width. Citation Information: Cancer Res 2013;73(24 Suppl): Abstract nr P2-18-18.

Toward an objective assessment of technical skills: a national survey of surgical program directors in Saudi Arabia.

BackgroundAfter almost a decade of implementing competency-based programs in postgraduate training programs, the assessment of technical skills remains more subjective than objective. National data on the assessment of technical skills during surgical training are lacking. We conducted this study to document the assessment tools for technical skills currently used in different surgical specialties, their relationship with remediation, the recommended tools from the program directors’ perspective, and program directors’ attitudes toward the available objective tools to assess technical skills.MethodsThis study was a cross-sectional survey of surgical program directors (PDs). The survey was initially developed using a focus group and was then sent to 116 PDs. The survey contains demographic information about the program, the objective assessment tools used, and the reason for not using assessment tools. The last section discusses the recommended tools to be used from the PDs’ perspective and the PDs’ attitude and motivation to apply these tools in each program. The associations between the responses to the assessment questions and remediation were statistically evaluated.ResultsSeventy-one (61%) participants responded. Of the respondents, 59% mentioned using only nonstandardized, subjective, direct observation for technical skills assessment. Sixty percent use only summative evaluation, whereas 15% perform only formative evaluations of their residents, and the remaining 22% conduct both summative and formative evaluations of their residents’ technical skills. Operative portfolios are kept by 53% of programs. The percentage of programs with mechanisms for remediation is 29% (19 of 65).ConclusionThe survey showed that surgical training programs use different tools to assess surgical skills competency. Having a clear remediation mechanism was highly associated with reporting remediation, which reflects the capability to detect struggling residents. Surgical training leadership should invest more in standardizing the assessment of surgical skills.

A curriculum to teach and evaluate resident skills in the management of postpartum hemorrhage

The aim of this study was the development of a curriculum for the management of postpartum hemorrhage and an objective assessment of technical skills (OSATS) for uterine compression sutures. Twenty-two residents participated in the study. Evaluations included the global rating scale, task-specific checklist, and pass/fail rating for the OSATS and task-specific checklist for the hemorrhage drill. The reliability and validity of the evaluation tools were calculated. The inter-rater reliability was 0.98 for the hemorrhage drill checklist, 0.77 for the global rating scale, and 0.93 for the uterine suture checklist. Construct validity evaluation revealed senior residents performed superiorly to junior residents on the global rating scale (P=0.006) and on the uterine suture checklist (P=0.04). There was an improvement in performance on the post-test in comparison to the pretest (P=0.001). We present an inexpensive, reliable, and valid curriculum to teach and evaluate the medical and surgical management of postpartum hemorrhage.

Objective Structured Assessment of Technical Skill in Upper Extremity Surgery

Objective assessment of technical skills in hand surgery has been lacking. This article reports on an Objective Structured Assessment of Technical Skills format of a multiple bench-station evaluation of orthopedic surgery residents' technical skills for 3 common upper extremity surgeries. Twenty-seven residents (6 postgraduate year [PGY] 2, 8 PGY 3, 8 PGY 4, and 5 PGY 5) participated in the examination. Each resident performed surgery on a cadaveric specimen at 3 stations, trigger finger release (TFR), open carpal tunnel release, and distal radius fracture fixation. A board-certified hand surgeon evaluated trainee performance at each station, using a procedure-specific detailed checklist, a validated global rating scale, and pass/fail assessment. A resident post-testing evaluation was collected. Construct validity with correlation between year in training and detailed checklist scores was demonstrated for TFR and carpal tunnel release; between year in training and global rating scores for TFR and distal radius fracture fixation; and between year in training and pass/fail assessment for TFR. Criterion validity was demonstrated by the correlation between global rating scale scores, detailed checklist scores, and pass/fail assessment for TFR, carpal tunnel release, and distal radius fracture fixation. Time to complete the surgery was not correlated with surgical performance. Residents rated the multiple-station Objective Structured Assessment of Technical Skills format as highly educational. This study reports that a surgeon's ability to release a trigger finger does not correlate specifically to his or her ability to perform a carpal tunnel release or to perform plate fixation of a radius fracture. The results of this study would indicate that, for 3 different surgical simulations representing procedures of varying complexity, assessments by a single assessment tool is not adequate. To completely understand a resident's abilities, assessment by checklist (understanding the steps of the surgery), global rating scales (assessment of basic surgical skills in light of lesser or greater complexity surgeries), and pass/fail assessment (examination of adverse events) are all necessary components. Therapeutic II.