Arthroscopic Surgical Skill Evaluation Tool Research Articles

Virtual reality simulation training improve diagnostic knee arthroscopy and meniscectomy skills: a prospective transfer validity study

PurposeLimited data exist on the actual transfer of skills learned using a virtual reality (VR) simulator for arthroscopy training because studies mainly focused on VR performance improvement and not on transfer to real word (transfer validity). The purpose of this single-blinded, controlled trial was to objectively investigate transfer validity in the context of initial knee arthroscopy training.MethodsFor this study, 36 junior resident orthopaedic surgeons (postgraduate year one and year two) without prior experience in arthroscopic surgery were enrolled to receive standard knee arthroscopy surgery training (NON-VR group) or standard training plus training on a hybrid virtual reality knee arthroscopy simulator (1 h/month) (VR group). At inclusion, all participants completed a questionnaire on their current arthroscopic technical skills. After 6 months of training, both groups performed three exercises that were evaluated independently by two blinded trainers: i) arthroscopic partial meniscectomy on a bench-top knee simulator; ii) supervised diagnostic knee arthroscopy on a cadaveric knee; and iii) supervised knee partial meniscectomy on a cadaveric knee. Training level was determined with the Arthroscopic Surgical Skill Evaluation Tool (ASSET) score.ResultsOverall, performance (ASSET scores) was better in the VR group than NON-VR group (difference in the global scores: p < 0.001, in bench-top meniscectomy scores: p = 0.03, in diagnostic knee arthroscopy on a cadaveric knee scores: p = 0.04, and in partial meniscectomy on a cadaveric knee scores: p = 0.02). Subgroup analysis by postgraduate year showed that the year-one NON-VR subgroup performed worse than the other subgroups, regardless of the exercise.ConclusionThis study showed the transferability of the technical skills acquired by novice residents on a hybrid virtual reality simulator to the bench-top and cadaveric models. Surgical skill acquired with a VR arthroscopy surgical simulator might safely improve arthroscopy competences in the operating room, also helping to standardise resident training and follow their progress.Level of evidence2

An investigation into gender bias in the evaluation of orthopedic trainee arthroscopic skills.

Women surgeons receive lower compensation, hold fewer academic positions, and hold fewer leadership positions than men, particularly in orthopedic surgery. Gender bias at the trainee level has been demonstrated in various surgical subspecialties, but there is a lack of information on gender bias within the orthopedic training environment. This study investigated whether implicit gender bias is present in the subjective evaluation of orthopedic trainee arthroscopic skills. After institutional review board approval, a web-based survey was sent to American Shoulder and Elbow Surgeons (ASES) members via the society's email listserve. Study participants were informed that the study was being done to develop a systematic evaluation method for trainees. The survey randomized participants to view and evaluate a prefellowship and a postfellowship video of "Rachel" (she/her) or "Charles" (he/him) performing a 15-point diagnostic shoulder arthroscopy. The videos for Rachel and Charles were identical except for the pronouns used in the video. Participants evaluated the trainee's skill level using questions from the Arthroscopic Surgical Skill Evaluation Tool (ASSET). Blinded and deidentified additional comments regarding the trainee's skill were classified as positive, negative, or neutral. Statistical analyses were used to compare scores and comments between Rachel and Charles. Of 1115 active ASES members, 181 ASES members started the survey and 106 watched both videos and were included in the analysis. Of the 106 participants completing the survey, 96 (91%) were men and 10 (9%) were women with a median (interquartile range) age of 44 (38-51). A teaching role was reported by 84 of 106 participants (79%). There was no significant difference between prefellowship scores (P=.87) or between postfellowship scores (P=.84) for the woman and man fellow. The numbers of comments classified as positive, negative, or neutral were not significantly different between the man and woman fellow (P=.19). Participants in teaching roles gave significantly lower scores to both fellows at both time points (P=.04), and participants who had fellow trainees were more likely to give negative comments to both fellows (P=.02). Trainee gender did not influence the ratings and comments participants gave for trainee arthroscopic skills, suggesting that gender bias may not play a major role in the evaluation of arthroscopic skill during orthopedic training.

Scoring metrics for assessing skills in arthroscopic rotator cuff repair: performance comparison study of novice and expert surgeons.

We aim to develop quantitative performance metrics and a deep learning model to objectively assess surgery skills between the novice and the expert surgeons for arthroscopic rotator cuff surgery. These proposed metrics can be used to give the surgeon an objective and a quantitative self-assessment platform. Ten shoulder arthroscopic rotator cuff surgeries were performed by two novices, and fourteen were performed by two expert surgeons. These surgeries were statistically analyzed. Two existing evaluation systems: Basic Arthroscopic Knee Skill Scoring System (BAKSSS) and the Arthroscopic Surgical Skill Evaluation Tool (ASSET), were used to validate our proposed metrics. In addition, a deep learning-based model called Automated Arthroscopic Video Evaluation Tool (AAVET) was developed toward automating quantitative assessments. The results revealed that novice surgeons used surgical tools approximately 10% less effectively and identified and stopped bleeding less swiftly. Our results showed a notable difference in the performance score between the experts and novices, and our metrics successfully identified these at the task level. Moreover, the F1-scores of each class are found as 78%, 87%, and 77% for classifying cases with no-tool, electrocautery, and shaver tool, respectively. We have constructed quantitative metrics that identified differences in the performances of expert and novice surgeons. Our ultimate goal is to validate metrics further and incorporate these into our virtual rotator cuff surgery simulator (ViRCAST), which has been under development. The initial results from AAVET show that the capability of the toolbox can be extended to create a fully automated performance evaluation platform.

A novel arthroscopy training program based on a 3D printed simulator

BackgroundArthroscopy is the most important and exciting contribution to sports medicine of the last 100 years. One of its main limitations, however, is the steep learning curve it requires, which is not easy to beat given the scarcity and the high price of arthroscopy simulators. Hypothesis/purposeTo describe and evaluate the effectiveness of an open-access arthroscopy training program based on a 3D-printed simulator. MethodsA model was designed, which was to be printed on a fused filament fabrication (FFF) 3D printer for home use with polylactic acid (PLA) filaments. Fourteen exercises were prepared, each with its timeframe and conceptual goals, arranged from least difficult to most challenging. Exercises were designed to assist subjects in developing the skills of an experienced arthroscopic surgeon through use of the simulator. Twenty subjects from nine hospitals completed the arthroscopy training program. Performance in each exercise was evaluated according to the Arthroscopic Surgical Skill Evaluation Tool (ASSET), taking into account the number of times a student had to repeat each exercise to complete it successfully. ResultsThe mean ASSET score for each exercise was 22 points (IQR 19–25) and the mean number of times students had to repeat each exercise was 16 (95% CI 15.27–17.97). Eighty-five percent of subjects completed the program. The device was printed without difficulty by an independent investigator without prior knowledge of 3D printing. The price of the device was under US$ 12. ConclusionSubjects exhibited an improvement in their basic arthroscopic skills on the simulator. The number of times each of them had to repeat each exercise was similar, which is indicative of a smooth progression of difficulty along the program. The device proved easy to print, accessible, economical, and effective. This is the first structured program based on an open-access arthroscopic simulator to obtain satisfactory results.

Construct Validity and Experience of Using a Low-cost Arthroscopic Knee Surgery Simulator

To validate an affordable and easily reproducible arthroscopic knee surgery simulator made from simple, low-cost materials by demonstrating its ability to distinguish experienced from novice arthroscopists (i.e., construct validity). Additionally, acceptance and usefulness of the simulator in medical training and education were assessed. The simulator was used to perform a partial meniscectomy in both menisci. External and intra-articular images obtained during the procedures were used to assess objective visual parameters. The Arthroscopic Surgical Skill Evaluation Tool and a Likert scale addressing individual perceptions about the simulator and its applicability in medical education were also used. The study was conducted at Hospital de Clínicas da Universidade Federal do Paraná, a teaching hospital in southern Brazil. Thirty sixth-year medical students were recruited, as well as 10 orthopedic surgeons who had knee arthroscopy expertise and were members of a sports traumatology and knee arthroscopy group. There were statistically significant differences between the groups in all objective parameters. The mean time to perform the procedure was 60% higher among students compared to surgeons. Students needed 72.5% more time on average to perform triangulations and obtained an error rate approximately twice higher in the assessment of the area removed from the menisci. Regarding Arthroscopic Surgical Skill Evaluation Tool scores, statistically significant differences were found between surgeons and students in all 8 domains and in the total score. The simulator was well accepted, as over 90% of participants found it useful for education and training, believed it contributed to teaching and assessing specific surgical steps and procedures, and reported that the task was enjoyable. The arthroscopic knee surgery simulator was largely accepted and had good applicability in objective measurement of surgical skills, distinguishing medical students from orthopedic surgeons and thus demonstrating construct validity.

Development of a certification examination for orthopedic sports medicine fellows

The purpose of this study was to develop a multifaceted examination to assess the competence of fellows following completion of a sports medicine fellowship. Orthopedic sports medicine fellows over 2 academic years were invited to participate in the study. Clinical skills were evaluated with objective structured clinical examinations, multiple-choice question examinations, an in-training evaluation report and a surgical logbook. Fellows’ performance of 3 technical procedures was assessed both intraoperatively and on cadavers: anterior cruciate ligament reconstruction (ACLR), arthroscopic rotator cuff repair (RCR) and arthroscopic shoulder Bankart repair. Technical procedural skills were assessed using previously validated task-specific checklists and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale. Over 2 years, 12 fellows were assessed. The Cronbach α for the technical assessments was greater than 0.8, and the interrater reliability for the cadaveric assessments was greater than 0.78, indicating satisfactory reliability. When assessed in the operating room, all fellows were determined to have achieved a minimal level of competence in the 3 surgical procedures, with the exception of 1 fellow who was not able achieve competence in ACLR. When their performance on cadaveric specimens was assessed, 2 of 12 (17%) fellows were not able to demonstrate a minimal level of competence in ACLR, 2 of 10 (20%) were not able to demonstrate a minimal level of competence for RCR and 3 of 10 (30%) were not able to demonstrate a minimal level of competence for Bankart repair. There was a disparity between fellows’ performance in the operating room and their performance in the high-fidelity cadaveric setting, suggesting that technical performance in the operating room may not be the most appropriate measure for assessment of fellows’ competence.

Orthopaedic Residents' Transfer of Knee Arthroscopic Abilities from the Simulator to the Operating Room.

The ultimate goal for any surgical simulation program is to prove the capability of transferring the skills learned to real-life surgical scenarios. We designed an arthroscopic partial meniscectomy (APM) training program and sought to determine its ability to transfer skills to real patients. Eleven junior orthopaedic residents and three expert knee surgeons were included. A low-fidelity knee simulator was used. Trainees had two baseline assessments of completing APM on a supervised real patient and on the simulator, measured using the Arthroscopic Surgical Skill Evaluation Tool (ASSET). After baseline, the trainees completed an APM training program and had a final evaluation of proficiency on the simulator and in real patients. Experts were also assessed for comparison. Statistical analysis was performed, assuming nonparametric behavior of variables. All trainees improved from a base score of 14 points in real patients and 10 points on the simulator to a final score of 39 points and 36 points, respectively (P < 0.01). The final trainee simulator score did not differ from experts on the simulator and was lower in real patients (36 versus 39 points, respectively, P ≤ 0.01), which resulted in a 92% transfer ratio for the simulator. Simulated training of APM in orthopaedic residents using a low-fidelity knee simulator proved to not only improve simulated proficiency but also successfully transfer skills to a real clinical scenario with a high model transfer ratio. Level II (Prospective Cohort Study).

The Effect of Triangulation Simulator Training on Arthroscopy Skills: A Prospective Randomized Controlled Trial

PurposeTo prospectively evaluate the transferability of skills acquired on a low-cost, at-home, nonanatomic triangulation simulation system to cadaveric models.MethodsWe randomized 28 medical students into either a simulator-training group (n = 14) or group with no training (control, n = 14). All subjects were pretested using a standardized checklist of arthroscopic skills on cadaveric knees and shoulders. Training-group subjects practiced on the triangulation simulator for 90 minutes per week for 4 consecutive weeks. Control subjects received no training. All subjects completed a post-test checklist of arthroscopic skills on cadaveric knees and shoulders, as well as 4 training tasks on the simulator. A blinded orthopaedic surgeon evaluated the arthroscopic videos using the Arthroscopic Surgical Skill Evaluation Tool (ASSET) score.ResultsTraining-group knee and shoulder ASSET scores increased from 12.2 ± 1.85 to 14.6 ± 2.76 (P = .02) and from 14.6 ± 3.5 to 17.9 ± 4.5 (P = .29), respectively. In the control group, knee and shoulder ASSET scores increased from 14.3 ± 3.12 to 14.25 ± 4.67 (P = .99) and from 14.2 ± 2.7 to 17.07 ± 6.7 (P = .58), respectively. There were no significant differences in the mean post-test ASSET scores between the training group and control group for either knee or shoulder arthroscopy. The post-test ASSET safety subscore during knee arthroscopy was significantly higher in the training group (P = .03). The training group was able to complete significantly more simulator tasks compared with controls (P = .003) at post-testing. A significant positive correlation was found between knee arthroscopy performance and the number of tasks completed during simulation post-testing (P = .043). There was no significant correlation between shoulder arthroscopy performance and simulation performance (P = .532).ConclusionsBasic triangulation skills may be acquired by training on a low-cost, at-home, nonanatomic triangulation simulation system, although the degree of transferability and universal joint applicability, as well as the existence of an early ceiling effect in skill development, could not be shown.Level of EvidenceLevel II, randomized controlled trial.

Impact of Simulation Training on Diagnostic Arthroscopy Performance: A Randomized Controlled Trial

PurposeTo determine the impact of training on a virtual reality arthroscopy simulator on both simulator and cadaveric performance in novice trainees.MethodsA randomized controlled trial of 28 participants without prior arthroscopic experience was conducted. All participants received a demonstration of how to use the ArthroVision Virtual Reality Simulator and were then randomized to receive either no training (control group, n = 14) or a fixed protocol of simulation training (n = 14). All participants took a pretest on the simulator, completing 9 tasks ranging from camera-steadying tasks to probing structures. The training group then trained on the simulator (1 time per week for 3 weeks). At week 4, all participants completed a 2-part post-test, including (1) performing all tasks on the simulator and (2) performing a diagnostic arthroscopy on a cadaveric knee and shoulder. An independent, blinded observer assessed the performance on diagnostic arthroscopy using the Arthroscopic Surgical Skill Evaluation Tool scale. To compare differences between non–normally distributed groups, the Mann-Whitney U test was used. An independent-samples t test was used for normally distributed groups. The Friedman test with pair-wise comparisons using Bonferroni correction was used to compare scores within groups at multiple time points. Bonferroni adjustment was applied as a multiplier to the P value; thus, the α level remained consistent. Significance was defined as P < .05.ResultsIn both groups, all tasks except task 5 (in which completion time was relatively fixed) showed a significant degree of correlation between task completion time and other task-specific metrics. A significant difference between the trained and control groups was found for post-test task completion time scores for all tasks. Qualitative analysis of box plots showed minimal change after 3 trials for most tasks in the training group. There was no statistical correlation between the performance on diagnostic arthroscopy on either the knee or shoulder and simulation training, with no difference in Arthroscopic Surgical Skill Evaluation Tool scores in the training group compared with controls.ConclusionsOur study suggests that an early ceiling effect is shown on the evaluated arthroscopic simulator model and that additional training past the point of proficiency on modern arthroscopic simulator models does not provide additional transferable benefits on a cadaveric model.Level of EvidenceLevel I, randomized controlled trial.

Knee Arthroscopy Simulation: A Randomized Controlled Trial Evaluating the Effectiveness of the Imperial Knee Arthroscopy Cognitive Task Analysis (IKACTA) Tool.

Virtual-reality and cadaveric simulations are expensive and not readily accessible. Innovative and accessible training adjuncts are required to help to meet training needs. Cognitive task analysis has been used extensively to train pilots and in other surgical specialties. However, the use of cognitive task analyses within orthopaedics is in its infancy. The purpose of this study was to evaluate the effectiveness of a novel cognitive task analysis tool to train novice surgeons in diagnostic knee arthroscopy in high-fidelity, phantom-limb simulation. Three expert knee surgeons were interviewed independently to generate a list of technical steps, decision points, and errors for diagnostic knee arthroscopy. A modified Delphi technique was used to generate the final cognitive task analysis. A video and a voiceover were recorded for each phase of this procedure. These were combined to produce the Imperial Knee Arthroscopy Cognitive Task Analysis (IKACTA) tool that utilizes written and audiovisual stimuli to describe each phase of a diagnostic knee arthroscopy. In this double-blinded, randomized controlled trial, a power calculation was performed prior to recruitment. Sixteen novice orthopaedic trainees who performed ≤10 diagnostic knee arthroscopies were randomized into 2 equal groups. The intervention group (IKACTA group) was given the IKACTA tool and the control group had no additional learning material. They were assessed objectively (validated Arthroscopic Surgical Skill Evaluation Tool [ASSET] global rating scale) on a high-fidelity, phantom-knee simulator. All participants, using the Likert rating scale, subjectively rated the tool. The mean ASSET score (and standard deviation) was 19.5 ± 3.7 points in the IKACTA group and 10.6 ± 2.3 points in the control group, resulting in an improvement of 8.9 points (95% confidence interval, 7.6 to 10.1 points; p = 0.002); the score was determined as 51.3% (19.5 of 38) for the IKACTA group, 27.9% (10.6 of 38) for the control group, and 23.4% (8.9 of 38) for the improvement. All participants agreed that the cognitive task analysis learning tool was a useful training adjunct to learning in the operating room. To our knowledge, this is the first cognitive task analysis in diagnostic knee arthroscopy that is user-friendly and inexpensive and has demonstrated significant benefits in training. The IKACTA will provide trainees with a demonstrably strong foundation in diagnostic knee arthroscopy that will flatten learning curves in both technical skills and decision-making.

Validation of a Dry Model for Assessing the Performance of Arthroscopic Hip Labral Repair

Background: Arthroscopic hip labral repair is a technically challenging and demanding surgical technique with a steep learning curve. Arthroscopic simulation allows trainees to develop these skills in a safe environment. Purpose: The purpose of this study was to evaluate the use of a combination of assessment ratings for the performance of arthroscopic hip labral repair on a dry model. Study Design: Cross-sectional study; Level of evidence, 3. Methods: A total of 47 participants including orthopaedic surgery residents (n = 37), sports medicine fellows (n = 5), and staff surgeons (n = 5) performed arthroscopic hip labral repair on a dry model. Prior arthroscopic experience was noted. Participants were evaluated by 2 orthopaedic surgeons using a task-specific checklist, the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task completion time, and a final global rating scale. All procedures were video-recorded and scored by an orthopaedic fellow blinded to the level of training of each participant. Results: The internal consistency/reliability (Cronbach alpha) using the total ASSET score for the procedure was high (intraclass correlation coefficient > 0.9). One-way analysis of variance for the total ASSET score demonstrated a difference between participants based on the level of training (F3,43 = 27.8, P < .001). A good correlation was seen between the ASSET score and previous exposure to arthroscopic procedures (r = 0.52-0.73, P < .001). The interrater reliability for the ASSET score was excellent (>0.9). Conclusion: The results of this study demonstrate that the use of dry models to assess the performance of arthroscopic hip labral repair by trainees is both valid and reliable. Further research will be required to demonstrate a correlation with performance on cadaveric specimens or in the operating room.

Performance Assessment of Arthroscopic Rotator Cuff Repair and Labral Repair in a Dry Shoulder Simulator

To evaluate the use of dry models to assess performance of arthroscopic rotator cuff repair (RCR) and labral repair (LR). Residents, fellows, and sports medicine staff performed an arthroscopic RCR and LR on a dry model. Any prior RCR and LR experience was noted. Staff surgeons assessed participants by use of task-specific checklists, the Arthroscopic Surgical Skill Evaluation Tool (ASSET), and a final overall global rating. All procedures were video recorded and were scored by a fellow blinded to the year of training of each participant. A total of 51 participants and 46 participants performed arthroscopic RCR and LR, respectively, on dry models. The internal consistency or reliability (Cronbach α) using the total ASSET score for the RCR and LR was high (>0.9). One-way analysis of variance for the total ASSET score showed a difference between participants based on year of training (P < .001) for both procedures. The inter-rater reliability for the ASSET score was excellent (>0.9) for both procedures. A good correlation was seen between the ASSET score and the year of training, as well as the previous number of sports rotations. The results of this study show evidence of construct validity when using dry models to assess performance of arthroscopic RCR and LR by residents. The results of this study support the use of arthroscopic simulation in the training of residents and fellows learning arthroscopic shoulder surgery.

Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation

The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model. Over 18months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review. The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching. As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.

Simulation of Anterior Cruciate Ligament Reconstruction in a Dry Model

Background: As the demand increases for demonstration of competence in surgical skill, the need for validated assessment tools also increases. Hypothesis/Purpose: The purpose of this study was to validate a dry knee model for the assessment of performance of anterior cruciate ligament reconstruction (ACLR). The hypothesis was that the combination of a checklist and a previously validated global rating scale would be a valid and reliable means of assessing ACLR when performed by residents in a dry model. Study Design: Controlled laboratory study. Methods: All residents, sports medicine staff, and fellows were invited to perform a hamstring ACLR using anteromedial drilling and Endobutton fixation on a dry model of an anterior cruciate ligament. Previous exposure to knee arthroscopy and ACLR was recorded. A detailed surgical manuscript and technique video were sent to all participants before the study. Residents were evaluated by staff surgeons with task-specific checklists created by use of a modified Delphi procedure and the Arthroscopic Surgical Skill Evaluation Tool (ASSET). Each procedure (hand movements and arthroscopic video) was recorded and scored by a fellow blinded to the year of training of each participant. Results: A total of 29 residents, 5 fellows, and 6 staff surgeons (40 participants total) performed an ACLR on the dry model. The internal reliability (Cronbach alpha) of the test when using the total ASSET score was very high (>0.9). One-way analysis of variance for the total ASSET score and the total checklist score demonstrated a difference between participants based on year of training (P < .001). Post hoc analysis demonstrated a significant difference in global ratings and checklist scores between junior residents (postgraduate year [PGY] 1-3) and senior residents (PGY 4 and 5) and senior residents and fellows (P < .05). A significant difference was seen between fellows and staff on the global rating (P < .05) but not on the checklist scores (P > .05). A good correlation was seen between the total ASSET score and prior exposure to knee arthroscopy (0.73) and ACLR (0.65). The interrater reliability (intraclass correlation coefficient) between the examiner ratings and the blinded assessor ratings for the total ASSET score was very high (>0.8). Conclusion: The results of this study provide evidence that the performance of an ACLR in a dry model is a reliable method of assessing a resident’s knowledge of the steps and instrumentation required, and the method shows evidence of validity. Clinical Relevance: These models may be used to ensure a minimal level of competence before residents and fellows perform ACLR in the operating room.

Validation Of A Dry Model For The Assessment Of Resident Performance Of Anterior Cruciate Ligament Reconstruction (ACLR)

Objectives:As the demand increases for demonstration of competence in surgical skill, the need for validated assessment tools also increases. The purpose of this study was to validate the use of a sawbones model for the assessment of performance of anterior cruciate reconstruction (ACLR) by residents. We hypothesized that the combination of a checklist and a previously validated global rating scale be a valid and reliable means of assessing ACLR when performed by residents in a dry model.Methods:All residents, sports medicine staff and fellows were invited to perform an ACLR on an ACL Sawbones model. Demographics regarding previous exposure to knee arthroscopy and ACLR were collected. All participants were asked to perform a hamstring ACLR using an anteromedial portal with Endobutton fixation on the femur - a detailed surgical manuscript and technique video was sent to all residents prior to the study. Residents were evaluated by faculty using a task-specific checklist created using a modified Delphi procedure, and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale. Each procedure was recorded, with videotaping of the hand movements and arthroscopic video recordings of the intra-articular procedure. These videos were scored by a fellow blinded to the year of training of each resident.Results:A total of 29 residents, six staff and five faculty performed an ACLR on the sawbones model (40 total). The overall reliability (Cronbach's Alpha) of the test using the total ASSET score was very high (>0.9). The reliability for the femoral checklist was 0.75, for the tibial checklist was 0.78, and 0.68 for the graft passage and fixation. One-way analysis of variance for the total ASSET score and the total checklist score demonstrated a difference between residents based upon year of training (p<0.001). Post hoc analysis demonstrated a significant difference in global ratings and checklist scores between junior residents (PGY1-3) and senior residents (PGY4&5), seniors and fellows, and fellows and staff (p<0.05). A good correlation was seen between the total ASSET score and prior exposure to knee arthroscopy (0.73) and ACLR (0.65). The inter-rater reliability (ICC) between faculty rating and blinded assessor for the total ASSET score was very high (>0.8).Conclusion:The use of a sawbones models to assess resident performance of ACLR using the ASSET global rating scale is valid and reliable. These models may be used to ensure a minimal level of competence prior to resident performance of ACLR in the operating room.

Assessing Improvement in Diagnostic Knee Arthroscopic Skill During the AAOS Fundamentals of Knee and Shoulder Arthroscopy Course for Orthopaedic Residents

Objectives:The purpose of the AAOS Fundamentals of Knee and Shoulder Arthroscopy Course for Orthopaedic Residents is to provide an opportunity for orthopaedic trainees to practice and improve basic arthroscopic surgical skills. During the course, residents spend the majority of their time in the Orthopaedic Learning Center laboratory performing basic arthroscopic procedures on cadaveric specimens under the direction of an expert faculty. Residents are also exposed to various didactic lectures during this intensive three-day course. Historically, feedback from the course has been extremely positive from both resident participants and faculty, however no attempt has previously been made to more objectively determine the degree to which improvement in resident skill has occurred. The purpose of this study is to determine whether participation in the AAOS Fundamentals of Arthroscopic Surgery Course for Residents led to improved diagnostic knee arthroscopy performance. We hypothesized that scores on post course assessments of knee arthroscopy skills and confidence would be higher in comparison to scores on the same assessments prior to the course.Methods:46 orthopaedic surgical residents enrolled in the AAOS Fundamentals of Arthroscopic Surgery Course volunteered to participate in the study. Participants were asked to perform a diagnostic knee arthroscopy on a cadaveric knee specimen both before and immediately after the conclusion of the knee portion of the course. Diagnostic knee arthroscopy performance was assessed using the Arthroscopic Surgical Skill Evaluation Tool (ASSET). Residents were also assessed before and after the course using the TolTech virtual reality simulator which enables the objective assessment of several measures that correlate with arthroscopic proficiency. Confidence, before and after the course, for several arthroscopic procedures was also assessed via a self-reported questionnaire.Results:Following the course, residents demonstrated a 9.2% improvement in mean ASSET score (p=0.001). ASSET score improved most (13.2%) for those with less than 20 arthroscopic cases (p=0.007). On the TolTech virtual reality simulator, mean scores demonstrated significant improvements for multiple measures of proficiency including: Time, which improved 18% (p=.01), Arthroscope path distance, which improved 13% (p=.02), Probe path distance, which improved 16% (p=.05), Smoothness, which improved 10% (p=.001), and excessive probe pressure, which improved 440% (p=.02). Overall completeness improved 1.8%, but this was not significant (p=0.11). Confidence in performing arthroscopic partial menisectomy improved 13% (p=.002) and confidence in performing ACL reconstruction improved 9% (p=.01).(Table 1).Conclusion:It was feasible to assess several outcomes related to resident proficiency and confidence within the structure of the AAOS Fundamentals of Arthroscopic Surgery Course. Resident participation in the course resulted in modest improvements for many of the outcomes measured demonstrating the value of this and similar courses. This data will be critically evaluated to determine curricular changes in an effort to maximize course benefit for future participants.

Use of an Objective Structured Assessment of Technical Skill (OSATS) after a Sports Rotation

Objectives:While the acquisition of competent technical skill is a defining characteristic of surgery, it is not measured systematically in residency. As all post-graduate medical training slowly shifts to a competency-based model, effective assessments of surgical and other technical skills after modules will become necessary. However, the best method for of assessing competence in technical skill in this setting is unknown, and is limited by both cost and access to resources. We hypothesized that a multi-station Objective Structured Assessment of Technical skill (OSATS), using sawbones models, would be a valid and reliable method of assessing resident competence in surgical skills after a sports medicine rotation.Methods:At the start of their three-month sports medicine rotation, each resident was provided a list of 10 surgical skills in which they were expected to demonstrate competence (Table 1). At the end of the rotation, each resident undertook an OSATS comprised of six randomly chosen stations - low-fidelity sawbones models were used in all stations. Residents were evaluated by faculty / staff surgeons using a previously validated global rating scale (the Arthroscopic Surgical Skill Evaluation Tool (ASSET)), as well as task-specific checklists created using a modified Delphi procedure, and a final five-point global rating scale (GRS) using the Drefus model of skill acquisition (1=novice, 2=advanced beginner, 3=competent, 4=proficient, 5=expert). All arthroscopic procedures were recorded, and all hand movements were videotaped - the videos were reviewed by a single, blinded observer, and correlation sought between the faculty ratings and the observer ratings.Results:Over 18 months, 27 residents (19 junior, 8 senior) sat the OSATS after their rotation, as well as seven sports medicine staff and seven fellows, for a total of 41 participants. The overall reliability of the OSATS as measured by Cronbach's Alpha was very high (0.9). A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each individual technical skill (p<0.001) - this difference was also seen for all stations. Post hoc analysis demonstrated a significant difference in the total ASSET score between junior (PGY1-3) and senior (PGY4&5) residents, senior residents and fellows, as well as between fellows and faculty (p<0.05)(Figure 1). A high correlation was seen between the faculty assessments and the blinded observer assessments for each station (>0.8).Conclusion:The results of this study demonstrate that an OSATS using dry models is a valid and reliable means of assessing technical skill in orthopaedic residents after a sports medical rotation. Interestingly, junior residents were not able to perform technical skills as well as senior residents despite an identical rotation, suggesting that overall surgical experience and exposure is as important as intensive teaching.