Task-specific Checklist Research Articles

Improving Resident Comfort With Peripartum Cesarean Hysterectomy Through the Use of Low-Fidelity Simulation Models.

Introduction Cesarean hysterectomy is a relatively rarely performed, complex, life-saving procedure considered during post-partum hemorrhage and other obstetric complications. This multi-site study aimed at validating a low-cost, low-fidelity cesarean hysterectomy model to support resident proficiency and increase their confidence in performing this critical procedure. Materials and methods We developed a low-fidelity, anatomically representative model for cesarean hysterectomy simulation purposes. Obstetrics and Gynecology residents at two residency programs were offered a simulation experience using the model, with a performance assessment via two validated assessment tools. The clinical case involved a 38 y/o woman with a delivery complicated by postpartum hemorrhage who failed conservative therapy and required a peripartum hysterectomy. Participants included 26 residents, postgraduate year (PGY)1 through PGY4, at two midwestern university-based obstetrics and gynecology residency programs. Results The median resident scores on the Task Specific Checklist (TSC) and Global Rating Scale (GRS) correlated with increasing PGY levels. The combined TSC+GRS score was a median of 40 out of 49 total for the PGY4 class, while the PGY1 class had a median combined TSC+GRS score of 12 of 49. The PGY2 and -3 classes had TSC+GRS scores of 14 and 28, respectively. The simulation model was well-received with a median 4/5 rating for improving comfort level with cesarean hysterectomy and a median 4/5 rating for model realism. Conclusion This study validated a cost-effective, easily reproducible model that highlights the vital anatomy relevant to a cesarean hysterectomy. The model and simulation offer a way to introduce cesarean hysterectomies to residents while in training, particularly at sites that may not perform a substantial number of these procedures.

Development and Evaluation of a Cesarean Hysterectomy Simulation Model

Introduction: Cesarean hysterectomy is a life-saving procedure considered during postpartum hemorrhage in cases of uterine atony refractory to management, placenta accreta spectrum, or other obstetric complications. The procedure has high morbidity and mortality despite the rising rates of worldwide and in the United States. A high percentage of OB/GYN residents graduate without preparedness to perform this procedure post-residency, citing insufficient residency exposure as the primary contributor. This study aims to use evaluation and assessment tools to validate a low-cost, low-fidelity cesarean hysterectomy model at multiple OB/GYN residency training sites across the United States to increase resident confidence and proficiency in performing this critical procedure. Materials and Methods: Model Design and Development: Gynecologic surgeons from two OB/GYN residencies reviewed the literature for cesarean hysterectomy education and simulation. A low fidelity, anatomically accurate model was developed for cesarean hysterectomy simulation purposes. Efforts were made to optimize for anatomic layout and realistic feel, particularly for the anatomic structures relevant to cesarean hysterectomy: uterine vessels, anterior and posterior peritoneum, relation of anterior uterus to the bladder and ureters. Study Design: The study received institutional board review exemption at the Loyola University Medical Center (LUMC) and the University of Toledo (UT). During the last two months of the academic calendar, OB/GYN residents were offered the opportunity to perform a cesarean hysterectomy on the model. Demographics were collected in addition to the number of cesarean and abdominal hysterectomies performed as well as the confidence in cesarean hysterectomies independently. A uniform set of surgical instruments with both appropriate and inappropriate tools were made available for residents to choose from. Throughout the simulation, residents were asked to verbalize each step being performed. One faculty rater at each institution was trained to use the objective and standardized assessment tools and had completed an inter- and intra-rater reliability assessment. All residents were evaluated using the same two objective assessment tools. The two validated assessment tools used to evaluate resident performance using the model were the Task Specific Checklist and the Global Rating Scale. The former assessed residents on a “performed or not” basis on 14 surgical steps specific to cesarean hysterectomy. The latter used a scale of 1 to 5, with higher values representing more advanced skill, to evaluate respect for tissue handling, time and motion, instrument handing, knowledge of instrument selection, operation flow, assistant use, and familiarity with the procedure. Results: A total of 26 residents, 18 of 19 at LUMC and 8 of 17 at UT, completed the cesarean hysterectomy simulation between May and June 2022. At LUMC, no graduating residents had performed a cesarean hysterectomy in their residency. At UT, graduating residents had performed up to 2. All of the residents, including PGY-4s, stated that they either disagree or strongly disagree with the statement “I feel confident performing cesarean hysterectomy independently.” As hypothesized, the median resident scores on the Task Specific Checklist and Global Rating scale correlated with increasing PGY level. The combined TSC+GRS score was a median of 40 out of 49 total for the PGY-4 class, while the PGY-1 class had a median combined TSC+GRS score of 12 of 49. The PGY-2 and -3 class had TSC+GRS scores of 14 and 28 respectively. The simulation model was well-received with a median 4/5 rating for improving comfort level with cesarean hysterectomy and a median 4/5 rating for model realism. Each uterine model cost $6.00 and took approximately 20 minutes to build. The optional base constructed for supporting the model, cost a one-time fee of $160. All materials used for construction were easily available online. Discussion: This study validated an affordable, easily reproducible model that highlights the most important anatomy relevant to a cesarean hysterectomy. The model was developed, produced, and tested at two residency programs with a total of 26 residents participating in the study. This is the first low-fidelity cesarean hysterectomy model inclusive of relevant surgical steps and offers near-real visceral feel that has been developed and assessed for validity. This multisite study created a realistic model with high reproducibility at a low cost. Limitations of this study included the fact that only 27 residents participated in the study from only two institutions. In addition, faculty evaluators were not blinded to the residents they rated. The model is a reliable and affordable way to introduce this procedure at residency programs, particularly those that do not perform a substantial number of cesarean hysterectomies.

Feedback in an Entrustment-Based Objective Structured Clinical Examination: Analysis of Content and Scoring Methods.

Background The integration of entrustable professional activities (EPAs) within objective structured clinical examinations (OSCEs) has yielded a valuable avenue for delivering timely feedback to residents. However, concerns about feedback quality persist. Objective This study aimed to assess the quality and content alignment of verbal feedback provided by examiners during an entrustment-based OSCE. Methods We conducted a progress test OSCE for internal medicine residents in 2022, assessing 7 EPAs. The immediate 2-minute feedback provided by examiners was recorded and analyzed using the Quality of Assessment of Learning (QuAL) score. We also analyzed the degree of alignment with EPA learning objectives: competency milestones and task-specific abilities. In a randomized crossover experiment, we compared the impact of 2 scoring methods used to assess residents' clinical performance (3-point entrustability scales vs task-specific checklists) on feedback quality and alignment. Results Twenty-one examiners provided feedback to 67 residents. The feedback demonstrated high quality (mean QuAL score 4.3 of 5) and significant alignment with the learning objectives of the EPAs. On average, examiners addressed in their feedback 2.5 milestones (61%) and 1.2 task-specific abilities (46%). The scoring methods used had no significant impact on QuAL scores (95% CI -0.3, 0.1, P=.28), alignment with competency milestones (95% CI -0.4, 0.1, P=.13), or alignment with task-specific abilities (95% CI -0.3, 0.1, P=.29). Conclusions In our entrustment-based OSCE, examiners consistently offered valuable feedback aligned with intended learning outcomes. Notably, we explored high-quality feedback and alignment as separate dimensions, finding no significant impact from our 2 scoring methods on either aspect.

Evaluating effectiveness of cadaveric arthroscopic training for orthopaedic residents: A comparison of joints and training levels.

This study aimed to evaluate the effects and interactions of training level and different joints on the outcomes of cadaveric arthroscopic training courses for orthopaedic residents. This prospective study enrolled 16 orthopaedic residents who voluntarily participated in a cadaveric training programme involving the shoulder, elbow, wrist, knee and ankle joints. Outcomes were quantitatively assessed using task-specific checklists and the Arthroscopic Surgery Skill Evaluation Tool. Two-way analysis of variance (ANOVA) was conducted to determine the significance of the interactions between joint and years of training. Resident scores significantly increased after the dedicated lectures in all five joints (p = 0.003 for the shoulder module, p < 0.001 for the other joints). Two-way ANOVA revealed that the progress made after the dedicated lectures was significantly impacted by the joint (p = 0.006) and training level × joint interaction (p = 0.005) but not by the training level (p = 0.47). The simple effect of the joint was examined using Sidak's multiple comparison test. Among junior residents, the dedicated lectures resulted in more substantial progress in elbow and wrist arthroscopy when compared to shoulder arthroscopy (p = 0.020 and p = 0.043, respectively). The results suggest that, in cadaveric arthroscopic training courses for orthopaedic residents, training outcomes are primarily impacted by the specific joint being trained rather than the training level. Specifically, junior residents demonstrated greater improvement with training in procedures that are less commonly encountered during on-the-job training, such as elbow and wrist arthroscopy. These findings suggest the need to prioritise wrist and elbow arthroscopic training for junior residents to optimise educational outcomes. Level III.

Assessing the efficacy of simulation-based education for paramedics in extended focused assessment with sonography for trauma under physician guidance

We investigated the effectiveness of simulation-based education in Focused Assessment with Sonography for Trauma (FAST) to increase the number of Emergency Medical Technicians (EMTs) capable of performing ultrasound examinations in vehicles under the guidance of a physician. Twenty-eight paramedics watched a 14-min video on the features of the ultrasound system, its use, and the scanning method for each part of the body. Each participant performed four FAST examinations using a portable ultrasound device, and the task performance was rated using the Task Specific Checklist (TSC) and Global Rating Scale (GRS). The time required for visualizing each examination site and each FAST was assessed. The mean time required for the first and fourth FAST was 144.6 ± 52.4 s and 90.5 ± 31.0 s, respectively. The time required for each test significantly decreased with repeated testing (p < 0.001). The time to complete FAST was significantly shortened for the pericardial cavity (33.4 ± 23.1/15.3 ± 10.6 s, p < 0.01), right thoracic cavity (25.2 ± 11.8/12.1 ± 8.3 s, p < 0.01), Morrison fossa (19.1 ± 10.8/10.8 ± 6.3 s, p < 0.05), and left thoracic cavity (19.0 ± 8.3/15.6 ± 8.3 s, p < 0.05). TSC and GRS scores were elevated, and all EMTs could obtain valid images. The combination of a brief video lecture and hands-on training significantly reduced the time required for FAST performance. Moreover, repeated practice enabled the EMTs to efficiently obtain accurate and clinically useful images.

657 Evaluating the Effectiveness of Virtual Reality for Arthroplasty Training in Orthopaedic Trainees: A Systematic Review

Abstract Introduction Virtual Reality (VR) provides high-fidelity simulation, enabling trainees to practice surgical procedures whilst reducing patient risk. The focus in orthopaedic literature has been on basic surgical skills and arthroscopic procedures. Research into arthroplasty simulation resources is under-represented. This systematic review assesses current literature on the effectiveness of VR for arthroplasty training. Method A systematic literature search was conducted across all relevant bibliographic databases. Randomised controlled trials (RCTs) meeting the inclusion criteria were selected and assessed by three independent reviewers. The primary outcome was the standardised mean difference (SMD) in either the global rating score or task-specific checklists at pre/post-intervention. This outcome was extracted using a piloted data extraction tool. Studies were assessed for risk of bias using a piloted risk of bias tool. Results 628 studies were identified, of which 624 did not meet the inclusion criteria. The four studies included in the meta-analysis displayed a trend that VR improved surgical competence more than existing methods (SMD 1.57, 95% CI [-0.45 to 3.59]; P = 0.13). Secondary outcomes in all studies showed VR improved trainee confidence and provided a more enjoyable experience. Conclusions VR improved surgical competency and improved trainee confidence in performing arthroplasty procedures, therefore indicating that this training modality could benefit both trainees and patients in the future. However, there is a paucity of evidence for VR in arthroplasty training, making results less precise and creating significant inter-study heterogeneity. This highlights the need for further high-quality RCTs to be produced assessing the effectiveness of VR for arthroplasty training.

Gathering validity evidence for a 3D-printed simulator for training of myringotomy and ventilation tube insertion.

This study aimed to gather validity evidence according to Messick's framework for a novel 3D-printed simulator for myringotomy with ventilation tube insertion for use in technical skills training of otorhinolaryngology (ORL) residents. The study included 15 junior ORL residents (trainees) and 13 experienced teaching otolaryngologists (experts). Experts and trainees first received an identically structured introduction to the procedure, simulator, and simulation setup. Five procedures performed by each participant were video-recorded and ordered randomly for blinded rating by two independent raters. The rating tools used were a global rating scale (GBRS) and a task-specific checklist. Validity evidence was collected according to Messick's framework. Differences in time consumption and performance scores were analyzed. Finally, a pass/fail standard was established using the contrasting groups' method. Trainees used significantly more time per procedure (109 s, 95% CI: 99-120) than experts (82 s, 95% CI: 71-93; p < .001). Adjusted for repetition and rater leniency, experts achieved an average GBRS score of 18.8 (95% CI: 18.3-19.2) out of 20 points, whereas trainees achieved an average of 17.1 points (95% CI: 16.6-17.5; p < .001). In contrast to the task-specific checklist, the GBRS score discriminated between repetition number and participant experience. The pass/fail standard for the GBRS was established at 18.4 points. We established educational validity evidence for a novel 3D-printed model for simulation-based training of ventilation tube insertion and established a reliable pass/fail standard. 1b.

Impact of Unexpected Death in a Simulation Scenario on Skill Retention, Stress, and Emotions: A Simulation-Based Randomized Controlled Trial.

Introduction The simulation of patient death remains controversial in simulation-based education. We investigated the effect of simulated patient death on learners' skill retention, stress levels, and emotions. Methods After ethics approval, we recruited residents at two Canadian universities. Participants were randomized to manage a simulated cardiac arrest ending with either the unexpected death (intervention group) or survival (control group) of the simulated patient (i.e., manikin). Three months later, all participants performed the same scenario but with the opposite outcome. Blinded video raters assessed participants' non-technical and technical crisis resource management (CRM) skills at both time points. Stress levels (represented by anxiety level, salivary cortisol concentration, and cognitive appraisal) and emotional valence were measured. Outcomes were analyzed using analysis of covariance (ANCOVA) or generalized estimating equations as appropriate. Results The analysis included 46 participants (intervention: n=24; control: n=22). Simulated death neither affected retention of non-technical CRM skills (mean retention Ottawa Global Rating Scale score in the death group [29.4, 95% CI: 27.0, 31.8] versus control group [29.4, 95% CI: 26.8, 32.0; p=0.87])nor technical CRM skills (mean retention task-specific checklist score in the manikin death group [11.8, 95% CI: 10.5, 13.0]versus the control group [12.5, 95% CI: 11.3, 13.7; p=0.69]). The simulated death had negative effects on participants' anxiety levels, cognitive appraisal, and emotions. Conclusion Simulated patient death did not affect the retention of non-technical or technical CRM skillsbut led to greater levels of short-term anxiety, stress, and negative emotions among learners.

Ultrasound improves percutaneous needle cricothyrotomy success in the novice anesthesia provider compared to the landmark technique

IntroductionThis study tested ultrasound guidance on the success of student registered nurse anesthetists’ performance of ultrasound guided percutaneous needle cricothyrotomy versus the superficial landmark approach. MethodsUsing a quasi-experimental crossover design, students were randomized to two groups to determine order of procedural performance: (1) superficial landmark, and (2) ultrasound guided percutaneous needle cricothyrotomy. All students completed both conditions at different time points. Performance was measured using a task-specific checklist survey tool and a global rating scale. Time to successful guidewire passage and number of attempts were measured along with accuracy. ResultsThe median scores for respect for tissue, time and motion, and use of assistant categories were significantly lower when using landmark-guidance compared to ultrasound-guidance to perform the procedure (P = .021; P = .042; P = .049). Using ultrasound guidance for the percutaneous needle cricothyrotomy resulted in significantly fewer number of attempts for cricothyroidotomy access (P < .001). Post survey data indicated overall, 87.6% (CI 95%: 7.86–9.31) of the students felt the use of ultrasound improved their experience and was a valuable tool in teaching percutaneous needle cricothyroidotomy (95.9%; CI95%: 8.56–9.77). Following the simulated experience, students felt more confident when performing the percutaneous needle cricothyrotomy procedure (87.4%; CI95%: 1.44–3.40). ConclusionsThe use of ultrasound guidance during percutaneous needle cricothyrotomy showed a decrease in the number of attempts by student nurse anesthetists’ when performing cricothyrotomy. Students found ultrasound guidance enhanced the ease, speed, confidence levels, and overall experience.

Effects of a structural simulation curriculum for the repair of obstetrical anal sphincter injuries.

To assess the short-term and skill-retention effects of a structural simulation curriculum for the repair of obstetrical anal sphincter injuries (OASIS). This prepost experimental study recruited 31 obstetrics and gynecology residents from Peking Union Medical College Hospital. OASIS repair skill evaluations (at all timepoints) comprised a task-specific checklist (TSC), global rating scale (GRS), and pass/fail grade. A beef-tongue model was adopted in the simulation curriculum. After baseline testing, residents completed a structured 1-h workshop. Two weeks later, a second test was performed, followed by a one-to-one teaching workshop, with the proper completion of a checklist as the end-point. Three months later, residents completed a third test. The overall pass rate at baseline was 16.1%, with higher GRS scores (p= 0.035) in senior residents. Additionally, previewing the procedure before class and a history of OASIS repair observation were associated with higher TSC and GRS scores. Significant skill improvement was observed in the second and third tests, with pass rates of 96.8% and 93.5% respectively. Both TSC and GRS scores were improved compared with baseline (p< 0.001). A structural simulation curriculum integrating deliberate practice under supervision improves OASIS repair competence and achieves satisfactory skill retention.

Impact of Rater Training on Residents Technical Skill Assessments: A Randomized Trial

The ACS/APDS Resident Skills Curriculum's Objective Structured Assessment of Technical Skills (OSATS) consists of task-specific checklists and a global rating scale (GRS) completed by raters. Prior work demonstrated a need for rater training. This study evaluates the impact of a rater-training curriculum on scoring discrimination, consistency, and validity for handsewn bowel anastomosis (HBA) and vascular anastomosis (VA). A rater training video model was developed, which included a GRS orientation and anchoring performances representing the range of potential scores. Faculty raters were randomized to rater training or no rater training and were asked to score videos of resident HBA/VA. Consensus scores were assigned to each video using a modified Delphi process (Gold Score). Trained and untrained scores were analyzed for discrimination and score spread and compared to the Gold Score for relative agreement. Eight general and eight vascular surgery faculty were randomized to score 24 HBA/VA videos. Rater training increased rater discrimination and decreased rating scale shrinkage for both VA (mean trained score: 2.83, variance 1.88; mean untrained score: 3.1, variance 1.14, p = 0.007) and HBA (mean trained score: 3.52, variance 1.44; mean untrained score: 3.42, variance 0.96, p = 0.033). On validity analyses, a comparison between each rater group vs Gold Score revealed a moderate training impact for VA, trained κ=0.65 vs untrained κ=0.57 and no impact for HBA, R1 κ = 0.71 vs R2 κ = 0.73. A rater-training curriculum improved raters' ability to differentiate performance levels and use a wider range of the scoring scale. However, despite rater training, there was persistent disagreement between faculty GRS scores with no groups reaching the agreement threshold for formative assessment. If technical skill exams are incorporated into high stakes assessments, consensus ratings via a standard setting process are likely a more valid option than individual faculty ratings.

Effect of Short-term-focused Training on a Phantom Model in Improving Operative Room Performance Among Surgical Residents: A Randomized Trial.

Meta-analysis has shown the effectiveness of various training methods for the acquisition of laparoscopic skills in surgical training. However, there is very limited literature focusing on the translation of skill acquisition on training models into improved operating room (OR) performance. This study was conducted to evaluate the effectiveness of the Tuebingen trainer with integrated Porcine tissue in improving OR the performance of surgical trainees using standard assessment tools. The study was a single-blinded double-armed randomized control study conducted between July 2016 and March 2018. Eighteen, fourth, and fifth semesters of surgery residents were included in the study. The baseline performance was assessed in OR by performing laparoscopic cholecystectomy using validated scores, that is, Global Operative Assessment of Laparoscopic Skills (GOALS), Additional Five Criteria, Task-specific Checklist, Error Checklist, Visual Analogue Scale. The residents were then randomized into trainee and nontrainee groups. The training group received 5 days of short-term-focused training on the Tuebingen trainer, and the improvement was reassessed in OR. The demographic profile of residents was similar. The baseline scores were comparable. The training group showed statistically significant improvement in GOALS (9.88±1.76 to 12±0.66, P=0.05 vs. 10.33±1.5 to 11.4±2.24, P=0.28), task-specific checklist (42.22±10.92 to 53.33±14.14, P=0.027 vs. 45.55±10.13 to 50±17.32, P=0.51), and error checklist. The operating time significantly reduced (36.0±4.03 vs. 50.44±11.39, P=0.0025) following training. Our study concludes that the training on the Tuebingen trainer with integrated porcine organs results in a statistically significant improvement in the OR performance of surgical residents as compared with the nontrained residents, thereby indicating a transfer of skills from training to OR.

Attending guidance advised: educational quality of surgical videos on YouTube.

YouTube is the most used platform for case preparation by surgical trainees. Despite its popular use, studies have noted limitations in surgical technique, safety, and vetting of these videos. This study identified the most viewed laparoscopic cholecystectomy (LC) videos on YouTube and analyzed the ability of attendings, residents, and medical students to identify critical portions of the procedure, technique, and limitations of the videos. An incognito search was conducted on YouTube using the term "laparoscopic cholecystectomy." Results were screened for length, publication date, and language. The top ten most viewed videos were presented to general surgery attendings, residents, and medical students at a single academic institution. Established rubrics were used for evaluation, including the Critical View of Safety (CVS) for LC, a modified Global Operative Assessment of Laparoscopic Skills (GOALS) score, a task-specific checklist, and visual analog scales for case difficulty and operator competence. Educational quality and likelihood of video recommendation for case preparation were evaluated using a Likert scale. Attending assessments were considered the gold standard. Six attending surgeons achieved excellent internal consistency on CVS, educational quality, and likelihood of recommendation scales, with Cronbach alpha (⍺) of 0.93, 0.92, and 0.92, respectively. ⍺ was ≥ 0.7 in all the other scales measured. Attending evaluations revealed that only one of the ten videos attained all three established CVS criteria. Four videos demonstrated none of the CVS criteria. The mean educational quality (mEQ) was 4.63 on a 10-point scale. The mean likelihood of recommendation (mLoR) for case preparation was 2.3 on a 5-point scale. Senior resident assessments (Postgraduate Year (PGY)4 + , n = 12) aligned with attending surgeons, with no statistically significant differences in CVS attainment, mEQ, and mLoR. Junior residents (PGY1-3, n = 17) and medical students (MS3-4, n = 20) exhibited significant difference with attendings in CVS attainment, mEQ, and mLoR for more than half the videos. Both groups tended to overrate videos compared to attendings. YouTube is the most popular unvetted resource used for case presentation by surgical trainees. Attending evaluations revealed that the most viewed LC videos on YouTube did not attain the CVS, and were deemed as inappropriate for case preparation, with low educational value. Senior resident video assessments closely aligned with attendings, while junior trainees were more likely to overstate video quality and value. Attending guidance and direction of trainees to high-quality, vetted resources for surgical case preparation is needed. This may also suggest a need for surgical societies with platforms for video sharing to prioritize the creation and dissemination of high-quality videos on easily accessible public platforms.

Mixed-Methods Analysis of a Validated Arthroscopic Knot-Tying Simulator With New Indirect Visualization Condition.

Background:Simulation provides low-risk opportunities for surgical trainees to learn and practice fundamental skills. One simulation tool for orthopaedics is the Arthroscopic Knot (ArK) Trainer, which has been validated as an effective simulation tool across multiple methodologies. Previous studies have investigated the ArK Trainer in its basic form using clear plexiglass, which allows direct visualization of tissue anchors.Purpose:Using a mixed-methods approach, we assessed and compared junior and senior trainees’ Seoul Medical Center (SMC) knot–tying performance under direct and indirect visualization.Study Design:Cross-sectional study.Methods:Fourteen orthopaedic surgery postgraduate trainees at a single medical school were recruited to participate. Trainees tied SMC knots using the Ark Trainer under direct and indirect visualization. A mixed-methods approach was used to evaluate knot-tying proficiency and characterize participants’ approach to knot-tying. Knot-tying proficiency was evaluated using validated tools: a task-specific checklist (TSC), a global rating scale (GRS), and a proficiency scale (PS). Participants’ approach to knot-tying was characterized using Likert-type questionnaires and semistructured interviews. An α level of .10 was set a priori owing to the small pool of trainees.Results:The 14 participants included 7 junior residents (postgraduate years [PGYs] 1 and 2) and 7 senior residents (PGY ≥3), of whom 3 were fellows (PGY 6). Senior trainees outperformed junior trainees on both versions of the ArK Trainer: clear (GRS, P = .055; PS, P = .075) and covered (TSC, P = .05). Overall, participants performed better under direct visualization conditions (GRS, P = .05). In semistructured interviews, significantly more senior trainees discussed relying on haptic cues while tying knots under direct visualization (P = .021). The majority of trainees agreed that both versions of the ArK Trainer were realistic and appropriate practice formats for their level of training.Conclusion:Senior trainees were significantly more experienced than were junior trainees in arthroscopic skill and outperformed them on both configurations: direct (PS and GRS) and indirect (TSC) visualization. Experienced trainees were significantly more likely to report using tactile cues to aid knot-tying under indirect visualization. It is likely that inexperienced trainees rely more heavily on direct visualization and that the use of tactile cues may be an indicator of knot-tying proficiency. Trainees recommended progression from direct to indirect visualization configurations for inexperienced learners.

Working Towards Competence: A Novel Application of Borderline Regression to a Task-Specific Checklist for Technical Skills in Novices

Demonstrated competence through frequent assessment is an expected goal for progressive development in competency-based medical education curricula. The Objective Structured Assessment of Technical Skill (OSATS) is considered a valid method of formative assessment, but in few instances have standards been set for determining competence. The present study used borderline regression methods to examine standard setting of performance on a complex technical task with novices assessed using an OSATS checklist. This was a single institution prospective single arm experimental design study. Participants were 58 non-medical undergraduate students with no previous surgical experience, who observed a computer-based training module on end-to-side vascular anastomosis. Subsequently, participants were provided two 20-minute training sessions, two weeks apart where they received expert feedback whilst performing the task on a low-fidelity model. After each training session, participants completed the task unaided. Sessions were recorded and assessed using an OSATS checklist retrospectively by experts. Paired t-test analyses indicate that for both the checklist total score (t(52) = 8.05, p < 0.001) and the global rating score (t(53) = 8.15, p < 0.001), individuals performed significantly better in Phase 2. Borderline regression analyses indicated that in Phase 1 (R2 = .60) and Phase 2 (R2 = .75), the OSATS checklist could adequately capture variation in performance in novices. Further, the checklist could reliably classify novices at three of the five global rating performance levels. Pass rates determined by regression equations improved from Phase 1 to Phase 2 on all global rating levels. With the increasing focus on competency-based medical education, it is imperative that training programs have the capacity to accurately assess outcomes and set minimum performance standards. Borderline regression methods can accurately differentiate novice learners of varying performance levels before and after training on a complex technical skill task using an OSATS checklist.

Development, translation, and preliminary validation of the neck dissection assessment tool.

The objective was to develop an assessment tool to evaluate residents' competency for neck dissection and provide preliminary evidence of feasibility, reliability, and validity. Six surgeons developed a neck dissection assessment tool using a modified Delphi method and evaluated 58 neck dissections from six junior and six senior otolaryngology residents. The assessment tool uses a double checklist: a previously validated global rating scale (GRS) and a task-specific checklist (TSC). Use of the instrument appeared feasible and the average scores on the GRS and TSC differed significantly between junior and senior residents. The Pearson correlation coefficient between both checklists was 0.87. Intraclass correlation (ICC) for inter-rater reliability was 0.69 for the GRS, and 0.80 for the TSC. This study provides preliminary evidence of feasibility, reliability, and validity for the first neck dissection assessment tool and provides a foundation for further psychometric analysis and research.

Implementation of the ACS/ APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory

BackgroundThe American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) Resident Skills Curriculum includes validated task-specific checklists and global rating scales (GRS) for Objective Structured Assessment of Technical Skills (OSATS). However, it does not include instructions on use of these assessment tools. Since consistency of ratings is a key feature of assessment, we explored rater reliability for two skills. MethodsSurgical faculty assessed hand-sewn bowel and vascular anastomoses in real-time using the OSATS GRS. OSATS were video-taped and independently evaluated by a research resident and surgical attending. Rating consistency was estimated using intraclass correlation coefficients (ICC) and generalizability analysis. ResultsThree-rater ICC coefficients across 24 videos ranged from 0.12 to 0.75. Generalizability reliability coefficients ranged from 0.55 to 0.8. Percent variance attributable to raters ranged from 2.7% to 32.1%. Pairwise agreement showed considerable inconsistency for both tasks. ConclusionsVariability of ratings for these two skills indicate the need for rater training to increase scoring agreement and decrease rater variability for technical skill assessments.

Objective Structured Assessment of One-Day Surgical Suture Training Using a New Interactive Training Model

Objective: Evaluate the acquired surgical suture skills in veterinary medicine students after one-day training using new designed interactive surgical suture training (ISST) model. Methodology: 150 students were subjected to one-day surgical suture training. An objective structured assessment of technical skills (OSATS) task-specific checklist of 20 tasks was performed by three faculty members. A questionnaire was completed by each student. Results: tasks like square knot using hand tie, appropriate tension of knotting, correct edge distance, and equal spacing between sutures were well performed by final year students significantly. For other tasks, the final year student got better scores without distinct differences (P > 0.05). More than 90% of students confirmed the easy use of the ISST model and it simulated the real training. They strongly recommended the ISST training model. Conclusions: one-day surgical suture training was very suitable for undergraduate students; also the ISST model was convenient and easy to use during training. The OSATS’s checklist and the questionnaire were a reasonable way of objective assessment of surgical suture training. We suggest training surgical sutures on the ISST model and using the OSATS checklist with a questionnaire for training evaluation to all medical students and trainees.

Multi-institutional Validation of a Vaginal Hysterectomy Simulation Model for Resident Training

The purpose of the research was to both develop a vaginal hysterectomy model with surgically pertinent anatomic landmarks and assess its validity for simulation training. A low-cost, reproducible vaginal hysterectomy model with relevant anatomic landmarks for key surgical steps. Nine academic and community-based obstetrics and gynecology residency programs. One hundred sixty-nine obstetrics and gynecology residents. A vaginal hysterectomy model with surgically pertinent anatomic landmarks was developed and tested for construct validity. Of the 184 available residents, 169 (91%) participated in this study and performed a vaginal hysterectomy procedure on the described model. The validated objective 7-item global rating scale (GRS) and the 13-item task-specific checklist (TSC) were used as tools to assess performance. The median TSC and GRS scores correlated with year of training, prior experience, and trainee confidence. In addition, the TSC scores also correlated with the GRS scores (p <.001) with regard to performance and resident year of training. Receiver Operator Curves for identification of the residents meeting national residency accreditation minimum numbers for vaginal hysterectomy using the GRS and TSC scores had an area under the curve of 0.89 and 0.83, respectively. This reduced-cost vaginal hysterectomy model offers high construct validity and pertinence for simulation.

Modification of Fundamentals of Laparoscopic Surgery Manual Skills Curriculum with Addition of a Vaginal Cuff Closure Task.

Study Objective Obstetrics and gynecology (OBGYN) residents are required to pass a Fundamentals of Laparoscopic Surgery (FLS) exam to qualify for written boards. FLS exam validity evidence has been questioned recently and concerns have been raised with its applicability to gynecologic surgical training. We present preliminary data from a pilot study which aims to correlate performance on FLS manual skills to a vaginal cuff closure model in a simulated setting. Design Prospective descriptive study Setting OBGYN residency program at a large urban academic medical center. Patients or Participants OBGYN residents in post-graduate year three. Interventions In addition to the standard five FLS tasks, participants are introduced to a previously studied vaginal cuff closure model during a teaching session. Before taking FLS exam, they are evaluated on a cuff closure using the modified Global Operative Assessment of Laparoscopic Skills (GOALS) and a task-specific checklist. After assessment, residents filled out a usability survey. Measurements and Main Results Seven residents participated between July and December 2019. The mean time to completion of the cuff closure was 14.6 minutes (SD= 2.2 minutes) and range of 12.5 to 18.5 minutes. In our sample size of 7, we did not detect a correlation between official composite FLS and vaginal cuff scores as measured with GOALS (rho = 0.1070, p = 0.819) or the checklist (rho = 0.359, p = 0.429). All residents found the cuff to be a useful addition to curriculum and preferred to replace an FLS task with the cuff model. We plan to continue enrollment of 11 residents per year for four years, to reach a sample size of 44. Conclusion In this preliminary analysis of a pilot study, we did not find a correlation between performance on a vaginal cuff task trainer and FLS manual skills. Residents preferred cuff closure to FLS tasks.