Assessment Of Surgical Trainees Research Articles

The influence of virtual reality simulation on surgical residents’ heart rate during an assessment of arthroscopic technical skills: A prospective, paired observational study

HypothesisTo demonstrate that a virtual reality (VR) simulation training program reduces heart rate variability during an assessment of surgical trainees’ technical skills in arthroscopy. Study designProspective observational matched study. Materials & methodsThirty-six orthopaedic surgery residents, new to arthroscopy, received standard training in arthroscopic knee surgery, supplemented by additional monthly training for 6months on a VR simulator for 16 of them. At inclusion, the 2 groups (VR and NON-VR) answered a questionnaire and performed a meniscectomy on a VR simulator. After 6months of training, two independent trainers blinded to the inclusion arms evaluated the technical skills of the two groups during meniscectomies on a model and on an anatomical subject. Heart rate variability (HRV) was measured using a wireless heart rate monitor during baseline, VR training, and assessment. ResultsAfter removing incomplete data, the analysis focused on 10 VR residents matched at inclusion with 10 NON-VR residents. The VR group had a significantly lower heart rate at the final assessment (p=0.02) and lower overall HRV (p=0.05). The low/high frequency ratio (LF/HF) was not significantly different between the groups (1.84 vs 2.05, p=0.66) but the before-after training comparison showed a greater decrease in this ratio in the VR group compared to the NON-VR group −0.76 (−41%) vs −0.08 (−4%). ConclusionThis study demonstrates a significant difference in heart rate variability between trained residents versus untrained residents during the final assessment of their technical skills at 6months. It appears that improving stress management should be an integral part of training programs in arthroscopic surgery. Clinical InterestVR simulators in arthroscopy could improve non-technical skills such as heart rate variability, from the perspective of accountability. Level of evidenceIII.

Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making

BackgroundVideos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment.MethodsA multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird’s-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment.Results and discussionThe scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio–video quality, and questionable feasibility for assessing readiness for practice.ConclusionVideo-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio–video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training.

The Successful Use of a Virtual Application to Enhance Formative Feedback- and Competency-Based Assessments in Surgical Training

INTRODUCTION: This study aimed to use a customized phone and computer application to improve formative feedback in surgery and assess competency over time. METHODS: General surgery residents at a single institution were enrolled in a prospective cohort study from August 2021 to February 2022. A survey was conducted at the beginning and conclusion of the study period to assess perceptions on feedback regarding technical skills. Afterwards, a customized phone application was used to assess competency using the Zwisch scale and a dictation feature. Salient features of the app included 15 core operations (appendectomy, cholecystectomy, hernia repair, laparotomy, etc) broken into 7-10 key steps, objective structured assessment of technical skills (OSATS), components that went well, and areas for improvement. RESULTS: Twenty-two surgery residents submitted 260 virtual feedback-forms over a six-month period. After the study, residents’ perception that they were receiving valuable, weekly feedback regarding technical skills increased from 27.2% to 100.0% (p < 0.001). The application also allowed the program to track competency over time and provide insight into critical areas for improvement (Fig. 1). Two of the most commonly dictated phrases by faculty included “make better use of your left hand,” and “remain aware of providing adequate counter-tension.” 90.9% of residents stated that the use of the virtual application enhanced their operative experience and would recommend its continued use for feedback and competency-based assessments.CONCLUSION: Phone applications greatly enhance formative, technical feedback for residents. These virtual applications can also be effectively used to assess improving competency over core operations in general surgery training.

SP2.1.11Real-world use of workplace-based assessments in surgical training: a UK nationwide exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training

Abstract Background The utility of Workplace-Based Assessments (WBAs) in surgical training depends on the way they are completed. This study aims to ascertain the real-world usage of WBAs, as perceived by UK surgical trainees. Methods An anonymous national online questionnaire was conducted via the Association of Surgeons in Training (ASiT). The evaluation employed mixed methods to interpret quantitative and qualitative data. Suggestions for improvement were identified from free-text comments and considered through a formal two-round modified Delphi consensus process by ASiT council members. Results Analysis included 906 complete responses from all surgical specialities across all training levels. WBA use was perceived to deviate markedly from the Joint Committee on Surgical Training standards with significantly identified misuse across all surgical specialities. Inaccurate completion was acknowledged by 89.6% of respondents, and some trainers appear complicit with 40.9% of respondents aware of 'unobserved sign-off', and 33.6% aware of 'password disclosure' by trainers. Most trainees felt the Annual Review of Competency Progression (ARCP) respected WBA quantity above quality (55.4%), and a third felt pressure to overstate the number completed (32.0%). Main reasons for misuse were time restraints, lack of engagement and the will to achieve career progression targets. Conclusions UK surgical trainees perceive a significant deviation from WBAs standards due to multiple factors. The frequency and nature of misuse are worrying and undermines the existing evidence for the WBAs role in training. We present consensus recommendations from ASiT for the improvement of WBA use in UK surgical training.

Feasibility and Evaluation of Surgical Simulation with Developed Crisis Scenarios: A Comparison of Performance by Vascular Surgery Training Paradigms

Surgical simulation is an integral component of training and has become increasingly vital in the evaluation and assessment of surgical trainees. Simulation proficiency determination has been traditionally based on accuracy and time to completion of various simulated tasks, but we were interested in assessing clinical judgment during a simulated crisis scenario. This study assessed the feasibility of creating a crisis simulator station for vascular surgery and evaluated the performance of vascular surgery integrated residents (0+5) and vascular surgery fellows (5+2) during a technical testing with an integrated crisis scenario. A Modified Delphi method was used to create vascular surgery crisis simulation stations containing a clinical scenario in conjunction with either an open or endovascular simulator. Senior level vascular surgery trainees from both integrated residencies (0+5) and traditional vascular surgery fellowships (5+2) were then evaluated on two simulation stations: 1) Elective carotid endarterectomy (CEA) where the crisis is a postoperative stroke and 2) Endovascular aneurysm repair (EVAR) for a ruptured abdominal aortic aneurysm (rAAA). Each simulation had a crisis scenario incorporated into the procedure. Assessment was completed using a performance assessment tool containing a Likert scale. Total score was calculated as a percentage. Scores were also sub-divided in the following four categories: Situation Recognition and Decision-making, Procedural Flow, Technical Skills, and Interpretation and Use of Imaging Skills. Student's t-test was used for analysis. 40 senior-level trainees were evaluated (27 fellows and 13 integrated residents) completing 80 simulations. The CEA crisis simulation yielded similar results between both groups (0+5 vs. 5+2, p = 1.00). The 0+5 residents in vascular surgery were graded to be more proficient in the EVAR for rAAA crisis simulation and demonstrated significant differences in Total Score (p = 0.04), Procedural Flow (p=0.03), and Interpretation and Use of Imaging Skills (p = 0.02). The creation of crisis-based simulation for trainees in vascular surgery is feasible and actionable. Integrated 0+5 residents performed similarly to 5+2 fellows on an open carotid endarterectomy (CEA) crisis simulation, but 0+5 residents scored significantly higher compared to traditional 5+2 fellows in an endovascular rAAA crisis simulation. Crisis simulation may offer better educational experiences and improved value compared to routine simulation. Further studies using different procedural models and clinical scenarios are needed to assess the validity of crisis simulation in vascular surgery and to better understand the performance disparities found between these training paradigms.

Real-world use of workplace based assessments in surgical training: A UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training

BackgroundDespite widespread uptake, the utility of Workplace Based Assessments (WBAs) is disputed and evidence underpinning their use is largely based upon their completion in ideal conditions, rather than the real-world setting. AimTo ascertain the real-world usage of WBAs, as perceived by UK surgical trainees. Materials and methodsAn anonymous online questionnaire conducted nationally via the Association of Surgeons in Training (ASiT). Evaluation of 906 completed trainee responses, across all surgical specialties and training levels, employed mixed methods to interpret quantitative and qualitative data. ResultsThe sample permitted a 3.0% confidence level with acceptable internal consistency (Cronbach's alpha 0.755). Formative use was supported by 72.5% and summative use was rejected by almost as many (66.3%). WBA use was perceived to deviate markedly from that recommended by the Joint Committee on Surgical Training (JCST). Significant misuse was identified and elements perceived as inaccurate appear commonplace across the breadth of surgical specialties. Inaccurate completion was acknowledged by 89.6% of respondents and some trainers appear complicit, 147 individuals (16.2%) having reported this to trainers, 40.9% aware of ‘unobserved sign-off’, and 33.6% aware of ‘password disclosure’ by trainers. Furthermore, a majority of trainees felt the Annual Review of Competency Progression (ARCP) respected WBA quantity above quality (55.4%), and a third felt pressure to overstate the number completed (32.0%). Reasons for misuse appeared largely centred upon time restraints, lack of engagement and a will to achieve the required targets for career progression. 1.5 ConclusionsThis study demonstrates that UK surgical trainees perceive that most trainees deviate from guidance in their use of WBAs. This is worrying in both the apparent frequency and nature of misuse and somewhat undermines existing evidence for their role in surgical training. Trainees perceive that required numbers of WBAs are too high, that training programmes fail to encourage their use as formative assessments, and that there is a lack of engagement by many trainees and trainers. We present consensus recommendations from ASiT for the improvement of WBA use in UK surgical training.

Video-Based Assessment in Surgical Education: A Scoping Review

Video-based assessment of residents' surgical skills may offer several advantages over direct observations of clinical performance in terms of objectivity, time-efficiency, and feasibility. Although video-based assessment is becoming more common in surgical training, a broad understanding of its utility is lacking. This scoping review explores video-based assessment in surgical training and presents the evidence supporting its use. A literature search was conducted using the Web of Science database with key words related to video-based assessment and surgical training. Exclusion criteria included articles not published in English and articles on undergraduate medical education, continuing professional development, or non-surgical disciplines. Initially, 702 articles were identified; after title, abstract, and full-text screening by two independent reviewers (SM and VM), 199 articles remained. We present the benefits of video-based assessment, including the ability to capture clinical ability in the operating room without decreasing intraoperative efficiency, as well as the potential to improve formative assessment and feedback practices. We describe the validity, reliability, and challenges of video-based assessment, as well as the use of video-based methods in clinical and simulated settings. We conclude by discussing questions that remain to be addressed. Although further research and cost-benefit analyses are required, greater adoption of video-based assessment into surgical training may help meet increased assessment demands in an era of competency-based medical education.

A snapshot of surgical resident training in Japan: results of a national-level needs assessment survey.

To evaluate the status of surgical training in Japan through a national-level needs assessment. A survey was sent to all 909 graduating residents (GRs) and their 611 program directors (PDs) for the year 2016. A working group of surgical educators from around the country was formed under the education committee of the Japan Surgical Society. The survey items were developed by consensus of this working group. The survey investigated the knowledge and problems of the current curriculum, and the status of the current residency training. The response rates were 56.3% of the GRs and 76.8% of the PDs. Among the participants, 47.6% of the GRs and 29.4% of the PDs believed that the residency curriculum did not match the clinical experience. Over 80% of the GRs and PDs agreed on the importance of training outside of the OR, whereas only 13% of the GRs had received such training regularly. Trainees also reported a lower satisfaction rate about the opportunity to train outside of the OR. This national-level needs assessment of surgical training in Japan identified several gaps in the curriculum. These results provide valuable data to assist the ongoing efforts for surgical residency curriculum improvement.

The evolution of surgical training in the UK.

Surgical training in the UK has undergone major reforms over the last few decades. The focus has shifted from time based training to competency based training programs. This paper discusses the transformation of assessment in surgical training in the UK from the apprenticeship model to a more objective workplace-based assessment model. The paper describes the different milestones during this transformation process and discusses the assessment of surgical and nonsurgical skills in a measurable way; moreover, it highlights the strengths and weaknesses of different assessment tools.

Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery.

With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge. We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels. We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved competitive accuracies of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within a 1-3 second window, without needing an observation of entire training trial. This study highlights the potential of deep architectures for efficient online skill assessment in modern surgical training.

The reliability of operative rating tool evaluations: How late is too late to provide operative performance feedback?

BackgroundOperative rating tools can enhance performance assessment in surgical training. However, assessments completed late may have questionable reliability. We evaluated the reliability of assessments according to evaluation time-to-completion. MethodsWe stratified assessments from MileMarker‘s™ Operative Entrustability Assessment by evaluation time-to-completion, using concordance correlation coefficient (CCC) between self-assessment and evaluator scores as a measure of reliability. ResultsOverall, self-assessment and evaluator scores were strongly correlated (CCC = 0.72; p < 0.001) though self-assessments were slightly higher (p = 0.048). Reliability remained stable for evaluations completed within 0 days (CCC = 0.77; p < 0.001), 1–3 days (CCC = 0.73; p < 0.001), and 4–13 days after surgery (CCC = 0.69; p < 0.001), but dropped for evaluations completed within 14–38 days (CCC = 0.60; p < 0.001) and over 38 days (CCC = 0.54; p < 0.001) after surgery. There was strong evidence for an interaction between time-to-completion and reliability (p < 0.001). ConclusionsOur data support the reliability of assessments completed until 2 weeks after surgery. This finding may help refine the interpretation of evaluation scores as surgical specialties move toward competency-based accreditation.

Development of a surgical trainer assessment questionnaire.

The assessment of surgical trainees has become well-established in practice; however, this is not yet the case for surgical trainers. The aim of this study was to develop a trainer assessment questionnaire (OxTrAQ) based on the 12 key trainer attributes identified by a previous systematic review, to then explore this questionnaire's potential domains and test its reliability. The OxTrAQ was piloted at two surgical training centres in two surgical sub-specialities. At each centre, every trainee assessed every trainer while under their clinical supervision. Internal consistency (Cronbach's alpha) was calculated for this 12-item questionnaire as 0.958. Factor analysis revealed two domains which accounted for 77.997% of the variance; these were the general domain (nine items) and the independence domain (three items). Inter-observer reliability was shown by the intraclass correlation coefficients (ICCs) (average measures) for the overall OxTrAQ score, general domain score and independence domain score which were 0.951, 0.945 and 0.955 for centre 1 and 0.857, 0.819 and 0.883 for centre 2, respectively. Intra-observer reliability was shown by the ICCs (average measures) for the overall OxTrAQ score, general domain score and independence domain score which were 0.925, 0.913 and 0.940, respectively. Construct validity was demonstrated by a high correlation (Spearman rho = 0.886, P = 0.019) between trainer scores and the scores predicted by the trainers themselves. The OxTrAQ is a simple trainer assessment questionnaire which has demonstrated both high inter- and intra-observer reliability.

The validity and reliability of workplacebased assessments in surgical training

Will we ever see WBAs as more than a box-ticking exercise?

Procedure-based assessments in trauma and orthopaedic training – The trainees’ perspective

Introduction: The study aimed to gain an understanding of the attitudes of trauma and orthopaedic (T&O) trainees regarding procedure-based assessments (PBAs) and identify factors that influence any perceived educational benefit.Methods and materials: A questionnaire was emailed to all T&O trainees in the UK via an established e-mail communication tool after an initial pilot exercise. The data were analysed using the online survey software.Results: Of the 616 trainees included 53% found PBAs useful as a learning tool for delivery of feedback. Trainees agreed that there were barriers to the successful use of PBAs (61%). Completing the PBA at the time of the procedure (p < 0.001) and the trainer delivering quality feedback with PBAs (p < 0.001) significantly increased the number of trainees perceiving an improvement in their practice. Completing higher numbers of PBAs did not have this effect (p = 0.26). There was wide geographical variation in the use of PBAs by trainees.Conclusions: This is the first nationwide study offering a deeper insight into factors influencing T&O trainees’ perceptions of the educational benefit gained from using PBAs. This study informs the debate on how to improve the effective use of PBAs in T&O training, and generally, of workplace-based assessments in surgical training.

What Evidence is There for the Use of Workplace-Based Assessment in Surgical Training?

Recent years have seen broad changes in postgraduate training with a move toward structured formative and summative evaluation of trainees' competencies using workplace-based assessment (WBA) tools. The fitness for purpose of these instruments in surgery has been much debated. The aim of this study is to explore the evidence underlying the introduction and ongoing use of WBAs in surgical training. A critical literature review was conducted to identify studies evaluating the use of WBAs in postgraduate surgical training. The search was conducted using the electronic databases PubMed for full-text articles in English. Additional critical evaluations of the curriculum relating to WBAs were included. The articles were synthesized in a narrative review. The implementation of WBA requirements in surgical training has occurred despite a relative dearth of direct evidence of their efficacy and benefit. Studies and critical reviews are being regularly undertaken to ensure that supporting evidence is accrued and the drive for improvement and refinement is maintained. It is emerging that WBAs are (contrary to their current nomenclature) formative tools for feedback and hence learning. They can facilitate the progression toward expert practice at the center of the zone of proximal development and the higher levels of Miller's pyramid, but fall short--owing to their focus on competence--of guiding surgical trainees to the higher levels of Maslow's hierarchy. Limited evidence has potentially undermined the introduction of WBAs in surgical training to date. There are misunderstandings regarding their use as either summative or formative educational tools. These shortcomings are an opportunity for further work in examining WBAs in their current or modified form.

Re: Workplace-based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme

Re: Workplace-based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme

Response to Re: Workplace-based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme

Response to Re: Workplace-based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme

Assessment of surgical trainees in the workplace

Assessment of surgical trainees in the workplace

Workplace‐based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme

The Intercollegiate Surgical Curriculum Programme was launched in the United Kingdom in 2007. At its heart was the reliance upon clear, defined curricula, competence-based training and the use of workplace-based assessments to assess the competence. The principle assessments used were Case-based Discussion, Procedure-based Assessments (PBA), Direct Observation of Procedural Skills, and Clinical Evaluation Exercise and a Multisource Feedback tool. We report the initial experience with that system, and most importantly, the experience with workplace-based assessment. Themes include issues around faculty development, misuse of assessments, inappropriate timing of assessments, concerns about validity and reliability of the assessments and concerns about the actual process of workplace-based assessments. Of the assessments, the PBA performed best. As a consequence, there has been an increased focus upon faculty development, while some of the assessments have been redesigned in line with the PBA. A global rating scale has been introduced that uses clinical anchors. The rating scales have also been altered with a reduction in the number of ratings while an enhanced description of the complexity of the case has been introduced within the Case-based Discussion and the Clinical Evaluation Exercise. A re-evaluation will take place in the near future.

Assessment methods in surgical training in the United Kingdom

A career in surgery in the United Kingdom demands a commitment to a long journey of assessment. The assessment methods used must ensure that the appropriate candidates are selected into a programme of study or a job and must guarantee public safety by regulating the progression of surgical trainees and the certification of trained surgeons. This review attempts to analyse the psychometric properties of various assessment methods used in the selection of candidates to medical school, job selection, progression in training, and certification. Validity is an indicator of how well an assessment measures what it is designed to measure. Reliability informs us whether a test is consistent in its outcome by measuring the reproducibility and discriminating ability of the test. In the long journey of assessment in surgical training, the same assessment formats are frequently being used for selection into a programme of study, job selection, progression, and certification. Although similar assessment methods are being used for different purposes in surgical training, the psychometric properties of these assessment methods have not been examined separately for each purpose. Because of the significance of these assessments for trainees and patients, their reliability and validity should be examined thoroughly in every context where the assessment method is being used.