Procedure-specific Checklist Research Articles

WHO cares? Safety checklists in echocardiography.

The number of potentially preventable medical errors that occur has been steadily increasing. These are a significant cause of patient morbidity, can lead to life-threatening complications and may result in a significant financial burden on health care. Effective communication and team working reduce errors and serious incidents. In particular the implementation of the World Health Organisation (WHO) Safe Surgery Checklist has been shown to reduce in-hospital mortality, postoperative complications and the incidence of surgical site infection. However an increasing number of complex medical procedures and interventions are being performed outside of the theatre environment. The lessons learnt from the surgical setting are relevant to other procedures performed in other areas. For the echocardiographer, transoesophageal echocardiography (TOE) is one such procedure in which there is the potential for medical errors that may result in patient harm. This risk is increased if patient sedation is being administered. The British Society of Echocardiography and the Association of Cardiothoracic Anaesthetists have developed a procedure specific checklist to facilitate the use of checklists into routine practice. In this article we discuss the evolution of the WHO safety checklist and explore its relevance to TOE.

Developing an Objective Structured Assessment of Technical Skills for Laparoscopic Suturing and Intracorporeal Knot Tying

To develop a teaching and assessment tool for laparoscopic suturing and intracorporeal knot tying. We designed an Objective Structured Assessment of Technical Skills (OSATS) tool that includes a procedure-specific checklist (PSC) and global rating scale (GRS) to assess laparoscopic suturing and intracorporeal knot-tying performance. Obstetrics and Gynecology residents at our institution were videotaped while performing a laparoscopic suturing and intracorporeal knot-tying task at a surgical simulation workshop. A total of 2 expert reviewers assessed resident performance using the OSATS tool during live performance and 1 month later using the videotaped recordings. OSATS scores were analyzed using the Wilcoxon rank-sum test. Data are presented as median scores (interquartile range [IQR]). Intrarater and interrater reliabilities were assessed using a Spearman correlation and are presented as an r correlation coefficient and p value. An r ≥ 0.8 was considered as a high correlation. After testing, we received feedback from residents and faculty to improve the OSATS tool as part of an iterative design process. In all, 14 of 21 residents (66.7%) completed the study, with 9 junior residents and 5 senior residents. Junior residents had a lower score on the PSC than senior residents did; however, this was not statistically significant (median = 6.0 [IQR: 4.0-10.0] and median = 13.0 [IQR: 10.0-13.0]; p = 0.09). There was excellent intrarater reliability with our OSATS tool (for PSC component, r = 0.88 for Rater 1 and 0.93 for Rater 2, both p < 0.0001; for GRS component, r = 0.85 for Rater 1 and 0.88 for Rater 2, both p ≤ 0.0002). The PSC also has high interrater reliability during live evaluation (r = 0.92; p < 0.0001), and during the videotape scoring with r = 0.77 (p = 0.001). Our OSATS tool may be a useful assessment and teaching tool for laparoscopic suturing and intracorporeal knot-tying skills. Overall, good intrarater reliability was demonstrated, suggesting that this tool may be useful for longitudinal assessment of surgical skills.

Diagnosing technical competence in six bedside procedures: comparing checklists and a global rating scale in the assessment of resident performance.

To compare procedure-specific checklists and a global rating scale in assessing technical competence. Two trained raters used procedure-specific checklists and a global rating scale to independently evaluate 218 video-recorded performances of six bedside procedures of varying complexity for technical competence. The procedures were completed by 47 residents participating in a formative simulation-based objective structured clinical examination at the University of Calgary in 2011. Pass/fail (competent/not competent) decisions were based on an overall global assessment item on the global rating scale. Raters provided written comments on performances they deemed not competent. Checklist minimum passing levels were set using traditional standard-setting methods. For each procedure, the global rating scale demonstrated higher internal reliability and lower interrater reliability than the checklist. However, interrater reliability was almost perfect for decisions on competence using the overall global assessment (Kappa range: 0.84-1.00). Clinically significant procedural errors were most often cited as reasons for ratings of not competent. Using checklist scores to diagnose competence demonstrated acceptable discrimination: The area under the curve ranged from 0.84 (95% CI 0.72-0.97) to 0.93 (95% CI 0.82-1.00). Checklist minimum passing levels demonstrated high sensitivity but low specificity for diagnosing competence. Assessment using a global rating scale may be superior to assessment using a checklist for evaluation of technical competence. Traditional standard-setting methods may establish checklist cut scores with too-low specificity: High checklist scores did not rule out incompetence. The role of clinically significant errors in determining procedural competence should be further evaluated.

Development and Validation of a High-Fidelity Porcine Laryngeal Surgical Simulator.

Design and validate a laryngeal surgical simulator to teach phonomicrosurgical techniques. Device development and prospective validation. Tertiary medical center. A novel laryngeal fixation device and custom laryngoscope were produced for use with ex vivo porcine larynx specimens. Vocal fold lesions such as nodules and keratotic lesions were simulated with silicone injections and epithelial markings. A prospective validation using postsimulation surveys, global rating scales, and procedure-specific checklists was performed with a group of 15 medical students, otolaryngology residents, fellows, and attending laryngologists. Three procedures were performed: vocal fold augmentation, excision of a simulated vocal fold nodule, and excision of a simulated vocal fold keratosis. Participants overwhelmingly agreed that the simulator provided a realistic dissection experience that taught skills that would transfer to real operating scenarios. Expert performance was statistically superior to novice performance for excision of simulated vocal fold nodules and keratotic lesions, while no difference was observed for injection laryngoplasty. The ability to learn and rehearse surgical procedures in a safe environment is invaluable, particularly for delicate and highly technical phonomicrosurgical operations. We have developed a high-fidelity laryngeal surgical simulator complete with pathological lesions such as nodules and keratoses to teach these procedures. A prospective study demonstrated validity of our global rating scale and checklist assessments for vocal fold nodule and keratosis excision procedures, allowing them to be confidently incorporated into phonomicrosurgical training programs for surgeons of all levels of expertise.

Development and content validation of performance assessments for endoscopic third ventriculostomy.

This study aims to develop and establish the content validity of multiple expert rating instruments to assess performance in endoscopic third ventriculostomy (ETV), collectively called the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT). The important aspects of ETV were identified through a review of current literature, ETV videos, and discussion with neurosurgeons, fellows, and residents. Three assessment measures were subsequently developed: a procedure-specific checklist (CL), a CL of surgical errors, and a global rating scale (GRS). Neurosurgeons from various countries, all identified as experts in ETV, were then invited to participate in a modified Delphi survey to establish the content validity of these instruments. In each Delphi round, experts rated their agreement including each procedural step, error, and GRS item in the respective instruments on a 5-point Likert scale. Seventeen experts agreed to participate in the study and completed all Delphi rounds. After item generation, a total of 27 procedural CL items, 26 error CL items, and 9 GRS items were posed to Delphi panelists for rating. An additional 17 procedural CL items, 12 error CL items, and 1 GRS item were added by panelists. After three rounds, strong consensus (>80% agreement) was achieved on 35 procedural CL items, 29 error CL items, and 10 GRS items. Moderate consensus (50-80% agreement) was achieved on an additional 7 procedural CL items and 1 error CL item. The final procedural and error checklist contained 42 and 30 items, respectively (divided into setup, exposure, navigation, ventriculostomy, and closure). The final GRS contained 10 items. We have established the content validity of three ETV assessment measures by iterative consensus of an international expert panel. Each measure provides unique assessment information and thus can be used individually or in combination, depending on the characteristics of the learner and the purpose of the assessment. These instruments must now be evaluated in both the simulated and operative settings, to determine their construct validity and reliability. Ultimately, the measures contained in the NEVAT may prove suitable for formative assessment during ETV training and potentially as summative assessment measures during certification.

MP14-03 VALIDATION OF AN INTRAOPERATIVE ASSESSMENT TOOL FOR EVALUATING TRAINEES AND PROVIDING FEEDBACK DURING ROBOT-ASSISTED RADICAL PROSTATECTOMY

INTRODUCTION AND OBJECTIVES: As urology training shifts toward competency-based frameworks, valid and reliable methods for high-stakes assessment of trainees are crucial. Standardized evaluation metrics are lacking for robot-assisted radical prostatectomy (RARP). As RARP is becoming the gold standard for treatment of localized prostate cancer, the development and validation of a RARP assessment instrument is timely. We are currently validating a RARP assessment tool for use in the intraoperative setting, with a primary focus on providing feedback throughout the learning curve. METHODS: An initial inventory of 13 procedural steps and 60 sub-steps was generated using a modified Delphi technique at McMaster University. We then recruited 13 RARP surgeons from across North America to serve as our expert Delphi panel. Experts anonymously rated each RARP step and sub-step on a 5-point Likert scale of agreement for inclusion in the final assessment tool. Qualitative feedback was elicited to determine appropriate step placement, wording, and suggestions. Responses were compiled, the inventory was edited through three iterations, and 100% consensus was achieved. Intraoperative da Vinci videos of residents were then recorded as they independently performed the steps of ‘dropping the bladder’ and ‘ligation of the dorsal venous complex’. Five videos each from PGY 3-5 residents and 5 randomly-selected expert videos from our database of RARP cases were edited to include these two steps. Videos have been distributed to our panel of 8 expert RARP surgeons who are serving as raters throughout the validation process. Raters are blinded to level of training and will use the inventory as a guide to assess each step using the validated GEARS tool (Goh, et al.), a procedure-specific checklist, and an overall ‘pass/fail’ rating. Raters will assess 2-4 steps at a time until all steps and sub-steps have been evaluated. RESULTS: Through the Delphi study, steps were decreased by 13% and a skip pattern was incorporated, and there was no attrition until the last round (final N1⁄412). The result was 13 critical steps with 52 substeps. The assessment tool will be piloted in our urology program and revisions will be made as necessary. CONCLUSIONS: Our team has developed the first comprehensive inventory of RARP steps with excellent expert consensus. A novel, psychometrically sound intraoperative RARP assessment tool that is currently undergoing validation will be incorporated into urology curricula. This instrument has the potential to be used for future credentialing of RARP surgeons.

Simulation in the Early Management of Gastroschisis

Our aim was to design, create, and validate a simulator model and simulation scenario for the early management of gastroschisis. Candidates of varying surgical experience had 1 attempt on an abdominal wall defect simulator and were scored for 4 different aspects: resuscitation of the neonate, application of a silo by both a global rating scale and a procedure-specific checklist, and nontechnical skills (scored by Non-Technical Skills scale). Surgical trainees subsequently received a focused teaching module on the resuscitative management and the surgical decision-making process, including bowel protection methods. Trainees then had a second attempt, which was objectively analyzed for improvement. Candidates attempted the simulation and were assessed, looking for construct validity. There was a statistically significant difference between candidate experience levels for all aspects of the simulation (resuscitation, global rating scale, procedure-specific checklist, and nontechnical skills) calculated using analysis of variance. Feedback forms gave us face validity, with a mean adjusted score of 8.3/10 for realism. After teaching the module, there was a statistically significant improvement (P < 0.05) of 20% for technical skills and 10% for nontechnical skills, which is comparable with similar controlled studies. We showed that creating and running a simulation scenario for the early management of gastroschisis is a feasible and useful tool for training and assessment. The simulation may also be able to discriminate between experience levels and could be used as a teaching aid to improve a surgeon's technical and nontechnical skills.

Wrong site surgery in otolaryngology–head and neck surgery

Wrong site surgery has received high public awareness this past decade, yet discussion specific to otolaryngology is limited. Literature review. We searched the MEDLINE database on PubMed from 1980 to 2013 and pursued the citations of key references further. We conducted a review of the literature and public patient safety reports on the scope, root causes, and prevention of wrong site surgery with emphasis on otolaryngology. A review of the literature reveals that otolaryngology procedures constitute 0.3% to 4.5% of all wrong site surgery events, and wrong site surgery accounts for 4% to 6% of all medical errors in otolaryngology. A significant proportion (9% to 21%) of otolaryngologists reported experiences with wrong site surgery over their career, and the events most frequently resulted in temporary injuries to the patient with few cases of permanent disability or death. Although otolaryngology procedures have similar root causes for wrong site events as other specialties, inverted imaging and ambiguity in site marking are particular challenges. Site-marking practices are variable among otolaryngologists, as it is not applicable to many otolaryngology procedures, yet these are common procedures that also constitute the majority of wrong site cases reported in otolaryngology. Future interventions to address these challenges related to otolaryngology-head and neck surgery might involve a standardized protocol to confirm imaging accuracy, a specialty- or procedure-specific checklist, a standardized alternative to site marking when marking is impractical, and other innovations. Evaluation of these interventions is becoming easier given the increasing mandatory reporting of these events that provides more reliable incidence data.

Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting

BackgroundCompetence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.MethodsThe study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.ResultsValidity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.ConclusionsWe designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.

Validation of a Global Assessment of Arthroscopic Skills in a Cadaveric Knee Model

The purpose of this study was to determine whether a global assessment of arthroscopic skills was valid for blinded assessment of cadaveric diagnostic knee arthroscopy. A global skills assessment for arthroscopy was created using a published theory of the development of expertise. Faculty surgeons, fellows, and residents were consented and enrolled in this institutional review board-approved validation study. All participants were oriented to the equipment and procedures for diagnostic arthroscopy of the knee. After reviewing the anatomic structures to be visualized, participants were allowed 10 minutes to complete a diagnostic arthroscopy of the knee. The hands and arthroscopic view were recorded during this attempt. Resident participants completed a second filmed diagnostic arthroscopy 1 week after the initial attempt. Five blinded reviewers watched the synchronized videos and assessed arthroscopic skills with a procedure-specific checklist and the newly developed global skills assessment. The agreement between reviewers was determined by intraclass correlation coefficient. Internal consistency was determined with Cronbach's α. Test-retest reliability was measured by correlating repeated arthroscopies by residents. The ability of the global assessment to discriminate skill levels was determined with between-group Mann-Whitney U tests. The agreement between global assessment scores was strong (I.C.C. = 0.80, 95% C.I. 0.68-0.92). The internal consistency of evaluations was excellent (Cronbach's α = 0.97), and the test-retest reliability was strong (r = 0.52). The global assessment score was shown to be able to discriminate between skill levels by an analysis of variance indicating the difference in means among the various levels of training (P < .0001). The Objective Assessment of Arthroscopic Skills is a useful adjunct to arthroscopic educators and learners and could be used for in-training evaluations. The Objective Assessment of Arthroscopic Skills is an instrument that can be employed to measure the impact of skills curricula, including but not limited to simulation.

Vacuum extraction: development and test of a procedure‐specific rating scale

To develop and validate an Objective Structured Assessment of Technical Skills (OSATS) scale for vacuum extraction. Two-part study design: Primarily, development of a procedure-specific checklist for vacuum extraction. Hereafter, validation of the developed OSATS scale for vacuum extraction in a prospective observational study. Rigshospitalet, University Hospital of Copenhagen. For development, an obstetric expert from each labor ward in Denmark (28 departments) was invited to participate. For validation, nine first-year residents and 10 chief physicians with daily work in the obstetric field were tested. The Delphi method was used for development of the scale. In a simulated vacuum extraction scenario, first-year residents and obstetric chief physicians were rated using the developed OSATS scale for vacuum extraction to test construct validity of the scale. Consensus for the content of the scale. To test the scale of Cronbach's alpha, interclass correlation and differential item function was calculated in the prospective study. 89% completed the first and 61% completed the second Delphi round. Hereafter, consensus was obtained. There was a significant difference between residents' and experts' performance for total score and for the score of the separate parts of the scale. Cronbach's alpha for total score and for the separate parts of the scale was 0.91-0.95 and interclass correlation 0.84-0.9. The OSATS scale for vacuum extraction is a reliable test for differentiating between competence levels in a simulated setting.

Objective structured assessment of technical skills and checklist scales reliability compared for high stakes assessments

The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA, UK, Australia and New Zealand. We use data from 19 applicants for higher surgical training in 2008 at the Royal College of Surgeons in Ireland to compare: (i) the objective structured assessment of technical skills (OSATS) method; and (ii) a procedure-specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons. The overall interrater reliability (IRR) of the OSATS assessment as determined by a correlation coefficient was 0.507 (P < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR. The checklist's overall IRR was 0.89. Individually, only five (26%) of the OSATS assessments reached the 0.8 level of IRR in contrast to 18 (95%) of the checklist assessments. We propose binary procedure-based assessment checklists as more reliable assessment instruments with more robust reproducibility.

A novel multimodal platform for assessing surgical technical skills

Background Established methods for assessing surgical performance face limitations. Global rating scales and procedure-specific checklists are resource intensive and rely on expert opinions. Alternatives that use technology to track hand movements, such as magnetic and optical tracking systems, are generally expensive and ill suited to the surgical environment. Methods The authors present a new platform that integrates a novel, low-cost optical tracking system, magnetic tracking technology and a videographic recording system to quantify surgical performance synchronously across all modalities. The validity of this platform was tested by examining its ability to differentiate between the performance of expert and novice participants on a basic surgical task. Results Each modality was able to differentiate between expert and novice participants, and metrics were well correlated across modalities. Conclusions The authors have developed a platform for assessing surgical performance. It can operate in the absence of expert raters and has the potential to provide immediate feedback to trainees.

Tu1468 Validation of a Novel Tool (Direct Observation of Polypectomy Skills-DOPyS) for Assessing Competency in Polypectomy

There is a void in the formal assessment of technical skills in polypectomy which is now considered an integral part of colonoscopy. It has been shown to reduce the incidence of colorectal cancer but does have associated complications. Polypectomy competency assessment should arguably be a part of the certification process for all endoscopists. A polypectomy competency assessment tool (Direct Observation of Polypectomy Skills (DOPyS), which incorporates a 33-point procedure-specific checklist and an overall competency score) has been developed and its reliability, examined.

Construct Validity of a Tool for Assessing Competency in Polypectomy

IntroductionThere is a void in the formal assessment of technical skills in polypectomy which is now considered an integral part of colonoscopy. It has been shown to reduce the incidence...

Toward Feasible, Valid, and Reliable Video-Based Assessments of Technical Surgical Skills in the Operating Room

To determine the feasibility, validity, inter-rater, and intertest reliability of 4 previously published video-based rating scales, for technical skills assessment on a benchmark laparoscopic procedure. Assessment of technical skills is crucial to the demonstration and maintenance of competent healthcare practitioners. Traditional assessment methods are prone to subjectivity through a lack of proven validity and reliability. Nineteen surgeons (6 novice and 13 experienced) performed a median of 2 laparoscopic cholecystectomies each (range 1-5) on 53 patients within 2 Academic Surgical Departments. All patients had a diagnosis of biliary colic. Surgical technical skills were rated posthoc in a blinded manner by 2 experienced observers on 4 video-based rating scales. The different scales used had been developed to assess generic or procedure-specific technical skills in a global manner, or on a procedure-specific checklist. Six of 53 procedures were excluded on the basis of intraoperative difficulty. Of the remaining 47 procedures, 14 were performed by 6 novice surgeons and 33 by the 13 experienced surgeons. There were statistically significant differences between performance of the 2 groups on the generic global rating scale (median 24 vs. 27, P = 0.031), though not on procedural or checklist-based scales. All scales demonstrated inter-rater reliability (alpha = 0.58-0.76), though only the global rating scales exhibited intertest reliability (alpha = 0.72). Video-based technical skills evaluation in the operating room is feasible, valid and reliable. Global rating scales hold promise for summative assessment, though further work is necessary to elucidate the value of procedural rating scales.