Procedure-specific Assessment Research Articles

Longitudinal Evaluation of Congenital Cardiovascular Surgical Performance and Skills Retention Using Silicone-Molded Heart Models.

Objective: Hands-on surgical training (HOST) for congenital heart surgery (CHS), utilizing silicone-molded models created from 3D-printing of patients' imaging data, was shown to improve surgical skills. However, the impact of repetition and frequency of repetition in retaining skills has not been previously investigated. We aimed to longitudinally evaluate the outcome for HOST on two example procedures of different technical difficulties with repeated attempts over a 15-week period. Methods: Five CHS trainees were prospectively recruited. Repair of coarctation of the aorta (CoA) and arterial switch operation (ASO) were selected as example procedures of relatively low and high technical difficulty. Procedural time and technical performance (using procedure-specific assessment tools by the participant, a peer-reviewer, and the proctor) were measured. Results: Coarctation repair performance scores improved after the first repetition but remained unchanged at the follow-up session. Likewise, CoA procedural time showed an early reduction but then remained stable (mean [standard deviation]: 29[14] vs 25[15] vs 23[9] min at 0, 1, and 4 weeks). Conversely, ASO performance scores improved during the first repetitions, but decreased after a longer time delay (>9 weeks). Arterial switch operation procedural time showed modest improvements across simulations but significantly reduced from the first to the last attempt: 119[20] versus 106[28] min at 0 and 15 weeks, P = .049. Conclusions: Complex procedures require multiple HOST repetitions, without excessive time delay to maintain long-term skills improvement. Conversely, a single session may be planned for simple procedures to achieve satisfactory medium-term results. Importantly, a consistent reduction in procedural times was recorded, supporting increased surgical efficiency.

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review.

There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity. A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis. A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes. Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

Development and preliminary validation of a new task-based objective procedure-specific assessment of inguinal hernia repair procedural safety.

Surgical videos coupled with structured assessments enable surgical training programs to provide independent competency evaluations and align with the American Board of Surgery's entrustable professional activities initiative. Existing assessment instruments for minimally invasive inguinal hernia repair (IHR) have limitations with regards to reliability, validity, and usability. A cross-sectional study of six surgeons using a novel objective, procedure-specific, 8-item competency assessment for minimally invasive inguinal hernia repair (IHR-OPSA) was performed to assess inter-rater reliability using a "safe" vs. "unsafe" scoring rubric. The IHR-OPSA was developed by three expert IHR surgeons, field tested with five IHR surgeons, and revised based upon feedback. The final instrument included: (1) incision/port placement; (2) dissection of peritoneal flap (TAPP) or dissection of peritoneal flap (TEP); (3) exposure; (4) reducing the sac; (5) full dissection of the myopectineal orifice; (6) mesh insertion; (7) mesh fixation; and (8) operation flow. The IHR-OPSA was applied by six expert IHR surgeons to 20 IHR surgical videos selected to include a spectrum of hernia procedures (15 laparoscopic, 5 robotic), anatomy (14 indirect, 5 direct, 1 femoral), and Global Case Difficulty (easy, average, hard). Inter-rater reliability was assessed against Gwet's AC2. The IHR-OPSA inter-rater reliability was good to excellent, ranging from 0.65 to 0.97 across the eight items. Assessments of robotic procedures had higher reliability with near perfect agreement for 7 of 8 items. In general, assessments of easier cases had higher levels of agreement than harder cases. A novel 8-item minimally invasive IHR assessment tool was developed and tested for inter-rater reliability using a "safe" vs. "unsafe" rating system with promising results. To promote instrument validity the IHR-OPSA was designed and evaluated within the context of intended use with iterative engagement with experts and testing of constructs against real-world operative videos.

Validity Evidence for Procedure-specific Competence Assessment Tools in Orthopaedic Surgery: A Scoping Review.

Competency-based training requires frequent assessment of residents' skills to determine clinical competence. This study reviews existing literature on procedure-specific competence assessment tools in orthopaedic surgery. A systematic search of eight databases up to May 2023 was conducted. Two reviewers independently assessed validity evidence and educational utility of each assessment tool and evaluated studies' methodological quality. Database searching identified 2,556 unique studies for title and abstract screening. Full texts of 290 studies were reviewed; 17 studies met the inclusion criteria. Bibliography review identified another five studies, totaling 22 studies examining 24 assessment tools included in the analysis. These tools assessed various orthopaedic surgery procedures within trauma, sports medicine, spine, and upper extremity. Overall validity evidence was low across all studies, and was lowest for consequences and highest for content. Methodological quality of studies was moderate. Educational utility assessment was not explicitly done for most tools. The paucity of current procedure-specific assessment tools in orthopaedic surgery lacks the validity evidence required to be used reliably in high-stake summative assessments. Study strengths include robust methodology and use of an evidence-based validity evidence framework. Poor-quality existing evidence is a limitation and highlights the need for evidence-based tools across more subspecialties.

A multi-national, video-based qualitative study to refine training guidelines for assigning an "unsafe" score in laparoscopic cholecystectomy critical view of safety.

The critical view of safety (CVS) was incorporated into a novel 6-item objective procedure-specific assessment for laparoscopic cholecystectomy (LC-CVS OPSA) to enhance focus on safe completion of surgical tasks and advance the American Board of Surgery's entrustable professional activities (EPAs) initiative. To enhance instrument development, a feasibility study was performed to elucidate expert surgeon perspectives regarding "safe" vs. "unsafe" practice. A multi-national consortium of 11 expert LC surgeons were asked to apply the LC-CVS OPSA to ten LC videos of varying surgical difficulty using a "safe" vs. "unsafe" scale. Raters were asked to provide written rationale for all "unsafe" ratings and invited to provide additional feedback regarding instrument clarity. A qualitative analysis was performed on written responses to extract major themes. Of the 660 ratings, 238 were scored as "unsafe" with substantial variation in distribution across tasks and raters. Analysis of the comments revealed three major categories of "unsafe" ratings: (a) inability to achieve the critical view of safety (intended outcome), (b) safe task completion but less than optimal surgical technique, and (c) safe task completion but risk for potential future complication. Analysis of reviewer comments also identified the potential for safe surgical practice even when CVS was not achieved, either due to unusual anatomy or severe pathology preventing safe visualization. Based upon findings, modifications to the instructions to raters for the LC-CVS OPSA were incorporated to enhance instrument reliability. A safety-based LC-CVS OPSA has the potential to significantly improve surgical training by incorporating CVS formally into learner assessment. This study documents the perspectives of expert biliary tract surgeons regarding clear identification and documentation of unsafe surgical practice for LC-CVS and enables the development of training materials to improve instrument reliability. Learnings from the study have been incorporated into rater instructions to enhance instrument reliability.

Laparoscopic cholecystectomy critical view of safety (LC-CVS): a multi-national validation study of an objective, procedure-specific assessment using video-based assessment (VBA).

A novel 6-item objective, procedure-specific assessment for laparoscopic cholecystectomy incorporating the critical view of safety (LC-CVS OPSA) was developed to support trainee formative and summative assessments. The LC-CVS OPSA included two retraction items (fundus and infundibulum retraction) and four CVS items (hepatocystic triangle visualization, gallbladder-liver separation, cystic artery identification, and cystic duct identification). The scoring rubric for retraction consisted of poor (frequently outside of defined range), adequate (minimally outside of defined range) and excellent (consistently inside defined range) and for CVS items were "poor-unsafe", "adequate-safe", or "excellent-safe". A multi-national consortium of 12 expert LC surgeons applied the OPSA-LC CVS to 35 unique LC videos and one duplicate video. Primary outcome measure was inter-rater reliability as measured by Gwet's AC2, a weighted measure that adjusts for scales with high probability of random agreement. Analysis of the inter-rater reliability was conducted on a collapsed dichotomous scoring rubric of "poor-unsafe" vs. "adequate/excellent-safe". Inter-rater reliability was high for all six items ranging from 0.76 (hepatocystic triangle visualization) to 0.86 (cystic duct identification). Intra-rater reliability for the single duplicate video was substantially higher across the six items ranging from 0.91 to 1.00. The novel 6-item OPSA LC CVS demonstrated high inter-rater reliability when tested with a multi-national consortium of LC expert surgeons. This brief instrument focused on safe surgical practice was designed to support the implementation of entrustable professional activities into busy surgical training programs. Instrument use coupled with video-based assessments creates novel datasets with the potential for artificial intelligence development including computer vision to drive assessment automation.

Using video-based assessment (VBA) to document fellow improvement in safely completing the jejunojejunostomy portion of laparoscopic Roux-en-Y gastric bypass (RYGB) surgery.

Surgical assessment instruments are used for formative and summative trainee evaluations. To characterize the features of existing instruments and a novel 12-item objective, procedure-specific assessment tool for Roux-en-Y Gastric Bypass (RYGB-OPSA), we evaluated the progress of a single surgical fellow over 17 consecutive surgeries. Seventeen consecutive RYGB videos completed between 8/2021 and 1/2022 by an academic hospital surgical fellow were de-identified and assessed by four board-certified bariatric surgeons using Global Operative Assessment of Laparoscopic Skills (GOALS), General Assessment of Surgical Skill (GASS), and RYGB-OPSA which includes the reflection of transverse colon, identification of ligament of Treitz, biliopancreatic and Roux limbs orientation, jejunal division point selection, stapler use, mesentery division, bleeding control, jejunojejunostomy (JJ) anastomotic site selection, apposition of JJ anastomotic site, JJ creation, common enterotomy closure of JJ, and integrity of anastomosis. The GASS measured economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, and achievement of hemostasis. RYGB-OPSA and GASS items were scored "poor-unsafe," "acceptable-safe," or "good-safe." Change in performance was measured by linear trendline slope. Over the course of 17 procedures, significant improvement was demonstrated by three GOALS items, GOALS overall score, GASS bimanual dexterity, and three RYGB-OPSA tasks: JJ creation, jejunal division point selection, and stapler use. Achievement of hemostasis declined but never rated "poor-unsafe." Overall RYGB-OPSA and GOALS trendlines documented significant increase across the 17 procedures. This examination of a bariatric surgery fellow's operative training experience as measured by three surgical assessment instruments demonstrated anticipated improvements in general skills and safe completion of procedure-specific tasks. Effective surgical assessment instruments have enough sensitivity to show improvement to enable meaningful trainee feedback (low-stakes assessments) as well as the ability to determine safe surgical practice to enable promotion to greater autonomous practice.

Assessment of Technical Competence in Distal Radius Fracture Fixation by a Volar Locking Plate: A Global Delphi Consensus Study

Volar locking plate fixation of distal radius fractures is a common orthopedic procedure and should be mastered by graduating orthopedic residents. Surgical education is transitioning from a traditional time-based approach to competency-based medical education. Valid and objective assessment is essential for successful transition. The purpose of this study was to develop a comprehensive, procedure-specific assessment tool to evaluate technical competence in volar locking plate osteosynthesis of a distal radius fracture. International orthopedic/trauma experts involved in resident education participated as panelists in a four-round online Delphi process to reach consensus on the content of the assessment tool. Round 1 was an item-generating round, in which the panelists identified potential assessment parameters. In round 2, the panelists rated the importance of each suggested assessment parameter and reached consensus on which to include in the assessment tool. Round 3 yielded specific assessment score intervals for specific bone and fracture models and is not reported in this study. In round 4, the panelists assigned weights to the assessment parameters on a 1-10 scale to determine how each parameter should have an impact on the overall results. Eighty-seven surgeons, representing 42 countries, participated in the study. Round 1 resulted in 45 assessment parameters, grouped into five procedural steps. After round 2, the number of parameters was reduced to 39. After the final round, an additional parameter was removed and weights were assigned to the remaining parameters. Using a systematic methodology, a preliminary assessment tool to evaluate technical competence in distal radius fracture fixation was developed. A consensus of international experts supports the content validity of the assessment tool. This assessment tool represents the first step in the evidence-based assessment essential for competency-based medical education. Before implementation, further studies exploring validity of variations of the assessment tool in different educational contexts are required.

Developing an Objective Structured Assessment of Technical Skills (OSATS) for Microlaryngoscopy.

Microlaryngoscopy is a basic technical skill in Oto-HNS. It is essential for residency programs to have a competency-based assessment tool to evaluate residents' performance of this procedure. An Objective Structured Assessment of Technical Skills (OSATS) is a procedure-specific assessment, which consists of the following: (a) Operation-Specific Checklist and (b) Global Rating Scale (GRS). The objective of this study was to create an OSATS for adult microlaryngoscopy. This was a prospective study, with an initial qualitative phase for OSATS development (Phase I), and a clinical pilot phase (Phase II). In Phase I, interviews were conducted with three laryngologists to establish a stepwise description of adult microlaryngoscopy and review a previously validated GRS for relevance to microlaryngoscopy. Responses were used to create a framework for the OSATS. The OSATS was then presented to Oto-HNS residents and laryngologists in an alternating fashion, for review of clarity and relevance. A pilot study was then performed to evaluate the resident performance of adult microlaryngoscopy. Multiple regression analysis was carried out to investigate whether training level, case complexity, and previous OSATS exposure could predict participant scores. Phase I of this study led to the creation of a 34-item OSATS. The pilot study (N=28 procedures) revealed that training level was significantly correlated with increased OSATS scores. There was no statistically significant correlation between case complexity and resident scores. Assessors reported the perceived utility of the OSATS and intent for use in residency training. Application of the proposed OSATS will allow for competency-based assessment of the resident performance of microlaryngoscopy. NA Laryngoscope, 133:2719-2724, 2023.

Validity of video-based general and procedure-specific self-assessment tools for surgical trainees in laparoscopic cholecystectomy.

IntroductionSelf-review of recorded surgical procedures offers new opportunities for trainees to extend technical learning outside the operating-room. Valid tools for self-assessment are required prior to evaluating the effectiveness of video-review in enhancing technical learning. Therefore, we aimed to contribute evidence regarding the validity of intraoperative performance assessment tools for video-based self-assessment by general surgery trainees when performing laparoscopic cholecystectomies.Methods and proceduresUsing a web-based platform, general surgery trainees in a university-based residency program submitted recorded laparoscopic cholecystectomy procedures where they acted as the supervised primary surgeon. Attending surgeons measured operative performance at the time of surgery using general and procedure-specific assessment tools [Global Operative Assessment of Laparoscopic Skills (GOALS) and Operative Performance Rating System (OPRS), respectively] and entrustability level (O-SCORE). Trainees self-evaluated their performance from video-review using the same instruments. The validity of GOALS and OPRS for trainee self-assessment was investigated by testing the hypotheses that self-assessment scores correlate with (H1) expert assessment scores, (H2) O-SCORE, and (H3) procedure time and that (H4) self-assessment based on these instruments differentiates junior [postgraduate year (PGY) 1–3] and senior trainees (PGY 4–5), as well as (H5)simple [Visual Analogue Scale (VAS) ≤ 4] versus complex cases (VAS > 4). All hypotheses were based on previous literature, defined a priori, and were tested according to the COSMIN consensus on measurement properties.ResultsA total of 35 videos were submitted (45% female and 45% senior trainees) and self-assessed. Our data supported 2 out of 5 hypotheses (H1 and H4) for GOALS and 3 out of 5 hypotheses (H1, H4 and H5) for OPRS, for trainee self-assessment.ConclusionsOPRS, a procedure-specific assessment tool, was better able to differentiate between groups expected to have different levels of intraoperative performance, compared to GOALS, a general assessment tool. Given the interest in video-based learning, there is a need to further develop valid procedure-specific tools to support video-based self-assessment by trainees in a range of procedures.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00464-022-09466-6.

Cleft Lip Repair Competence Can Be Evaluated with High-fidelity Simulation.

Background:Demonstrating competency before independent practice is increasingly important in surgery. This study tests the hypothesis that a high-fidelity cleft lip simulator can be used to discriminate performance between training levels, demonstrating its utility for assessing procedural competence.Methods:During this prospective cohort study, participants performed a unilateral cleft lip repair on a high-fidelity simulator. Videos were blindly rated using the Objective Structured Assessment of Technical Skills (OSATS) and the Unilateral Cleft Lip Repair Competency Assessment Tool (UCLR). Digital measurement of symmetry was estimated. Influence of training level and cumulative prior experience on each score was estimated using Pearson r.Results:Participants (n = 26) ranged from postgraduate year 3 to craniofacial fellow. Training level correlated best with UCLR (R = 0.4842, P = 0.0122*) and more weakly with OSATS (R = 0.3645, P = 0.0671), whereas cumulative prior experience only weakly correlated with UCLR (R = 0.3450, P = 0.0843) and not with OSATS (R = 0.1609, P = 0.4323). UCLR subscores indicated marking the repair had little correlation with training level (R = 0.2802, P = 0.1656), whereas performance and result did (R = 0.5152, P = 0.0071*, R = 0.4226, P = 0.0315*, respectively). Correlation between symmetry measures and training level was weak.Conclusions:High-fidelity simulation paired with an appropriate procedure-specific assessment tool has the construct validity to evaluate performance for cleft lip repair. Simply being able to mark a cleft lip repair is not an accurate independent assessment method nor is symmetry of the final result.

Validity Evidence for Procedure-Specific Competency Assessment Tools in Cardiovascular and Thoracic Surgery: A Scoping Review

Surgical education has shifted from a time-based approach to the achievement and demonstration of procedural competency. High quality, objective assessment instruments are required to support this new approach. This study comprehensively reviewed the literature to identify and evaluate available procedure-specific assessment instruments in cardiothoracic and vascular surgery. A systematic search of 8 databases identified studies containing procedure-specific operative assessment instruments in cardiothoracic and vascular surgery. Generic global rating scales were excluded, unless modified to be procedure-specific. Two reviewers independently evaluated the validity evidence, methodological rigour and educational utility of each instrument using objective scoring criteria. Validity evidence was evaluated with a scoring tool aligned with the contemporary framework of validity. Methodological rigour was evaluated using the Medical Education Research Study Quality Instrument. Educational utility was evaluated according to the Accreditation Council for Graduate Medical Education (ACGME) framework. There were 2130 unique studies describing procedure-specific assessment in surgery. Of these, 9 studies evaluating 8 procedure-specific assessment instruments met inclusion criteria for cardiothoracic and vascular surgery. Four instruments were identified in thoracic surgery, 2 in cardiac surgery, and 2 in vascular surgery. Only 1 instrument was designed to evaluate surgeon performance, with the remainder designed to evaluate residents. No single instrument scored the maximum score of 15 for validity evidence. The highest score was 11, with 62.5% (n = 5) of instruments scoring greater than 10. All tools attained high scores in content validity, with minimal evidence generally presented regarding the consequences of assessment using a particular instrument. All but 1 instrument scored greater than 11 out of a maximum 16.5 points for methodological rigour. Very few studies reported on the ACGME domains of educational utility. In an era where surgical education is shifting towards the demonstration of procedural competency, objective procedure-specific assessment is critical. This review identified that few procedure-specific assessment instruments in cardiothoracic and vascular surgery exist, emphasizing the need for such instruments to ensure the success of competency-based education models.

Assessment of surgical competence for neck dissection: a pilot study

Background:Progressive implementation of the milestone competence-based curriculum has created a need for new objective and validated means to assess resident surgical proficiency. A previous systematic review of the literature by our group has highlighted a shortage of tools assessing surgical competence in oncologic procedures in otolaryngology — head and neck surgery.Methods:We developed a procedure-specific assessment tool for neck dissection using a modified Delphi method. The 2-part design was modelled on the previously validated Objective Structured Assessment of Technical Skills checklist. The tool was then validated through a 1-year multicentric prospective study in collaboration with the residents and faculty from our academic centre. Additionally, we developed an online survey to assess the acceptability by residents and staff before and after the validation studies.Results:A total of 29 evaluations were completed throughout the 2016–2017 academic year. Acceptability ranked high for both residents and staff, with a single discrepancy in responses regarding a potential formative as opposed to summative use of the tool. Validation study results showed significantly higher checklist scores among senior residents than junior residents, as well as a significant score progression over time (p < 0.05). Trends in scores on the task-specific tool correlated highly to results obtained on a validated global rating scale (p < 0.05).Conclusion:The first tool assessing surgical competence in oncologic otolaryngology — head and neck surgery has been developed and shows promising validity.

Development and validation of an objective scoring tool to evaluate surgical dissection: Dissection Assessment for Robotic Technique (DART).

Evaluation of surgical competency has important implications for training new surgeons, accreditation, and improving patient outcomes. A method to specifically evaluate dissection performance does not yet exist. This project aimed to design a tool to assess surgical dissection quality. Delphi method was used to validate structure and content of the dissection evaluation. A multi-institutional and multi-disciplinary panel of 14 expert surgeons systematically evaluated each element of the dissection tool. Ten blinded reviewers evaluated 46 de-identified videos of pelvic lymph node and seminal vesicle dissections during the robot-assisted radical prostatectomy. Inter-rater variability was calculated using prevalence-adjusted and bias-adjusted kappa. The area under the curve from receiver operating characteristic curve was used to assess discrimination power for overall DART scores as well as domains in discriminating trainees (≤100 robotic cases) from experts (>100). Four rounds of Delphi method achieved language and content validity in 27/28 elements. Use of 3- or 5-point scale remained contested; thus, both scales were evaluated during validation. The 3-point scale showed improved kappa for each domain. Experts demonstrated significantly greater total scores on both scales (3-point, p< 0.001; 5-point, p< 0.001). The ability to distinguish experience was equivalent for total score on both scales (3-point AUC= 0.92, CI 0.82-1.00, 5-point AUC= 0.92, CI 0.83-1.00). We present the development and validation of Dissection Assessment for Robotic Technique (DART), an objective and reproducible 3-point surgical assessment to evaluate tissue dissection. DART can effectively differentiate levels of surgeon experience and can be used in multiple surgical steps.

Development and Validation of a Vaginal Anterior Repair Simulation Model for Surgical Training.

The aims of this study were to design and to validate a surgical simulation model for vaginal anterior colporrhaphy. Obstetrics and gynecology residents were divided into two groups: "novice" and "advanced." "Expert" group comprised female pelvic medicine and reconstructive surgery (FPMRS) fellows and faculty. All subjects were recorded performing anterior colporrhaphy using the simulation model. Deidentified, silent videos of recorded performances were evaluated by a blinded reviewer using the Objective Structured Assessment of Technical Skills (OSATS) and a procedure-specific assessment, based on the American Board of Obstetrics and Gynecology milestone evaluation. Analysis of variance was used for statistical comparisons. Thirty-seven participants included 13 novices, 18 advanced residents, and six experts. The experts scored significantly higher than both resident groups in each of the domains of the OSATS and the procedure assessment. The mean OSATS score for experts was 32.30 (±6.06) (maximum, 35); novice, 17.15 (±5.84); and advanced, 21.11 (±5.61) (P = 0.001). The experts' scores for the procedure assessment (maximum, 25) was 23.00 (±3.95); novice, 10.62 (±4.70); and advanced, 14.33 (±4.73) (P < 0.001). The two trainee groups did not differ significantly in their overall performance. Most trainees (29 [94%]) evaluated the model as a useful tool in learning this procedure. This low-fidelity model can be constructed easily using common materials. The model cost is $43.92 with reusable components ($11 per use). Higher scores by experts versus trainees confirm construct validity of this model. It may serve as a valuable simulation tool allowing trainees to improve their surgical technique for anterior colporrhaphy.

Development and validation of a procedure-specific assessment tool for hands-on surgical training in congenital heart surgery

BackgroundHands-on surgical simulation has been sought to address training limitations within congenital heart surgery (CHS). However, there is a need for objective assessment methods to measure surgeons’ performance to justify its global adoption. This study aimed to validate a procedure-specific assessment tool for the simulation of the arterial switch operation on 3D-printed models and to evaluate the consistency of scoring among evaluators with different levels of experience in CHS. MethodsFive “expert” and 5 “junior” surgeons performed the arterial switch procedure on 3D-printed models with transposition of the great arteries during 2 hands-on surgical training courses. Their performance was retrospectively assessed by 9 evaluators with varying experience in CHS (staff surgeons, resident surgeons, and non-MD raters). Assessments were done using 2 assessment tools: the Hands-On Surgical Training–Congenital Heart Surgery (HOST-CHS) assessment tool and the global rating scale (GRS). ResultsThe HOST-CHS tool showed a higher interrater and intrarater reliability compared with the GRS. Total scores for expert surgeons were highly consistent across all evaluators. Non-MD raters’ total scores for junior surgeons were slightly higher than those of residents and staff evaluators. All grades of evaluator were able to discriminate between junior and expert surgeons. ConclusionsThis study demonstrates the development and validation of an objective, procedure-specific assessment tool for the arterial switch operation with consistency among evaluators with different experience. There is now a platform for quantifying and accurately evaluating performance, which will be highly beneficial in training and developing the next generation of congenital heart surgeons.

Quantitative assessment of technical performance during hands-on surgical training of the arterial switch operation using 3-dimensional printed heart models

ObjectiveData supporting the use of hands-on simulation in congenital heart surgery are promising but primarily qualitative. This study aimed to demonstrate if there was an objective improvement in time and technical performance of the arterial switch procedure on 3-dimensional printed heart models by surgeons using a validated assessment method. MethodsA total of 30 surgeons of varying experience performed the arterial switch procedure twice on 3-dimensional printed models with transposition of the great arteries during the Hands-on Surgical Training courses. Surgeons' performances were recorded and retrospectively assessed for both time and performance using the Hands-on Surgical Training-Congenital Heart Surgery tool, a validated procedure-specific assessment tool for the arterial switch. ResultsA total of 60 videos were scored. Eighty percent of surgeons (24/30) had improved from their first attempt. The mean total score of the first attempt performance compared with the second was 103 and 120, respectively, with a mean difference in score of 17 (95% confidence interval, 10-24). All surgeons were statistically significantly quicker in their second attempt. The mean time for the first attempt compared with the second was 1 hour, 28 minutes, 4 seconds and 1 hour, 5 minutes, and 45 seconds, respectively, with a mean difference of 0 hours, 22 minutes, 19 seconds (95% confidence interval, 0 hours, 15 minutes, 22 seconds to 0 hours, 25 minutes, 34 seconds). ConclusionsThis is the first study to demonstrate an objective improvement in time and technical performance of the arterial switch procedure on 3-dimensional printed heart models. This supports the evidence that simulation in the form of deliberate practice with constructive, objective feedback is fundamental in the training of future congenital heart surgeons. These simulations and assessments should be incorporated to create structured, standardized training curricula within congenital heart surgery.

Ensuring Competency in Simulated Open Aortic Aneurysm Repair Using a Newly Developed Procedure-specific Assessment Tool- A Reliable Tool with Credible Pass-fail Standard for Assessment During Training, Testing and Certification

Introduction: There is an increasing need to establish structured simulation-based training programs in open repair of infrarenal aortic aneurysms (AAA) to address the current challenges such as work hour restrictions and decreasing frequency of open surgical repairs with the widespread shift to endovascular treatment (1,2). Simulation-based training programs for open AAA repair require a dedicated assessment tool with a credible pass-fail score to ensure minimum competences before apprenticeship on patients. Previous studies have used existing surgical assessment tools (3,4), however an up-to-date, and procedure-specific tool covering all steps of open AAA repair has yet to be developed and validated using a contemporary framework. The aims of the study were to develop a procedure-specific assessment tool for simulated open AAA repair, gather validity evidence for the tool, and establish a pass/fail standard for assessment. Methods: Validity evidence was explored using the contemporary framework by Messick (5). Three vascular surgeons with expertise in open AAA repair, and a professor in medical education with expertise in assessment and validation, developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) to ensure content evidence. Vascular surgical trainees with varying experience were invited to perform an open AAA repair on an abdominal model for open aortic surgery with pulsatile flow. All participants received e-learning materials prior to the operations. To ensure the quality and integrity of data, all procedures were performed in a standardized simulation-based environment with the presence of a supervisor who provided guidance when needed. The procedures were video recorded, anonymised, and scored independently by three expert vascular surgeons in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass-fail standard to determine consequences of testing. Results: A total of 16 novices and nine experienced open vascular surgeons participated in the study. The OPERATE consists of twelve items including four global scores (Figure 1). A high internal consistency for all items (Cronbach's alpha .92) and a high inter-rater reliability (Cronbach's alpha .94) were achieved. The novices performed significantly inferior from the experienced group with mean OPERATE scores of 15.1 and 25.6 (p=0.01), respectively (Figure 2). The pass/fail standard based on the Angoff method was 27.7. One novice passed the test while six experienced failed to achieve the pass-fail score. Conclusion: Strong validity evidence with a credible pass-fail standard was established for the newly developed procedure-specific assessment tool OPERATE, demonstrating its reliable use for structured formative assessment during training and summative assessment during testing or certification. The OPERATE has high discriminatory abilities and a high pass-fail standard emphasising the need for novice vascular surgeons to train in a simulation-based environment to achieve the minimum competences and skillsets before apprenticeship training on patients. The low scores in the experienced group underlines that familiarisation with the simulation equipment must be ensured before performance is assessed.Figure 2Box and whiskers plot showing the scores of the two groups.View Large Image Figure ViewerDownload Hi-res image Download (PPT) Disclosure: Nothing to disclose

P121 ESOPHAGECTOMY-SPECIFIC OBJECTIVE STRUCTURED ASSESSMENT OF TECHNICAL SKILL (E-OSATS): VALIDATION PROTOCOL

Abstract Aim The objective of this study is to develop the ‘Esophagectomy-specific Objective Structured Assessment of Technical Skills’ tool (E-OSATS) and determine its validity, reliability and feasibility of use. Background &amp; Methods Minimally invasive esophagectomy (MIE) is a complex and technically demanding procedure. Increased morbidity and mortality are associated with the long learning curve of the procedure. Specific and structured feedback may shorten the learning curve and decrease the associated morbidity and mortality. Consensus regarding the essential procedural steps has recently been achieved and allows for development and validation of a procedure specific assessment tool for MIE (E-OSATS). The E-OSATS was created by anchoring previously defined essential steps for both Ivor-Lewis (IL) and McKeown (McK) approach. Ten video recordings of novice surgeons (&lt;15 MIE performed) and 10 recordings of expert surgeons (&gt;100 MIE performed) should be rated by two independent, blinded and trained raters. Records of time needed for the assessment are created. Inter-rater and test-retest reliability will be determined. Validity will be determined by analyzing differences between novice and expert surgeons. Results All IL video recordings are collected, McK collection is still pending. First assessments of video recordings are expected to take place in August 2019. We aim to collect all assessments before December 2019. Conclusion We hypothesize that we will demonstrate the feasibility, reliability and validity of the E-OSATS in MIE specific assessment of technical skill. This tool will allow for specific and structured feedback and will potentially shorten the learning curve and decrease learning associated morbidity.

Implementation and evaluation of a comprehensive proficiency-based curriculum in an advanced, minimally invasive procedure: amulti-institutional Canadian experience.

A comprehensive proficiency-based curriculum for an advanced, minimally invasive procedure was previously developed and shown to be more educationally effective than conventional surgery training. To implement and evaluate this proficiency-based curriculum in 2 academic general surgery residency programs in Canada. Two academic, general surgery residency programs at university hospitals in Ontario, Canada. An 8-week, proficiency-based curriculum consisted of a didactic component (lectures, small group sessions, assigned readings) and a simulation-based component (proficiency-based training in laparoscopic enteroenterostomy and a simulated operating room crisis scenario). It was offered to postgraduate years 2-5 general surgery residents in 2 academic programs in Canada. Pre- and postcurriculum procedure-specific knowledge and psychomotor skills were assessed using a 25-item knowledge test and a procedure-specific assessment scale. Postcurriculum nontechnical skills were assessed using the Nontechnical Skills for Surgeons scale. Participants' perceptions about the curriculum were assessed using a questionnaire. Direct costs for curriculum implementation were recorded. Twenty-five residents participated in the curriculum across 2 programs. Completion of the curriculum resulted in significant improvement in technical skills (45 [37.5-65] versus 88 [85-93]; P< .01) and demonstration of "acceptable" situational awareness (3 [3-4]), decision-making (3[3-4]), teamwork and communication (3 [2-4]), and leadership (3 [3-4]) skills. There was no improvement in procedure-specific knowledge (48 [40-64] versus 58 [48-60]; P = .39). Participants perceived all components of the curriculum as educationally valuable, and 96% agreed and/or strongly agreed that this curriculum should continue to be a part of academic curriculum. The average cost of curriculum implementation was $613.05 Canadian dollars per participant. Lack of faculty supervision was the main barrier to implementation with only 65% of participants agreeing and/or strongly agreeing that quantity of faculty supervision was optimal. A comprehensive proficiency-based curriculum for an advanced, minimally invasive procedure was successfully implemented and evaluated at 2 academic general surgery residency programs in Canada. Adequate faculty preceptor resources are essential for widespread implementation.