Technical Performance Score Research Articles

Abstract 113: Technical Performance Scores as a quality assessment tool in Congenital Heart Surgery; preliminary analysis of validation data

BACKGROUND : In previous work from our institution, individual practitioner technical performance of surgical procedures has been shown to be an important contributor to outcome and resource utilization in selected congenital cardiac operations. We have developed a Technical Scoring System for evaluation of majority of congenital cardiac procedures and wish to validate its efficacy as a self assessment tool for quality improvement. METHODS : All patients who were discharged between Jan 1 2011 and Dec 31, 2011 were included in this study. Based on discharge echocardiograms, a technical performance score was assigned using previously designed criteria. Case complexity was determined by RACHS-I category, postoperative adverse events and mortality were prospectively monitored. Outcomes were analyzed by non parametric methods. RESULTS : There were a total of 842 discharges encompassing all ages from neonates to adults. 560 (67%) were in RACHS-1 categories 1 to 3 (low risk) and 130 (15%) were in RACHS-1 categories 4 to 6 (high risk) and 152 (18%) could not be categorized. Technical performance scores were as follows: 436 (52%) class 1-optimal, 242(29%) class 2-adequate, 96 (115) class 3-inadequate, and 68(8%) could not classified. Occurrence of major adverse events, mortality and length of stay were all significantly higher in class 3. CONCLUSION : Preliminary data validates the usefulness of the technical scoring system as a quality assessment tool.

687: The relationship of type of feedback to learning during simulation: a randomized clinical trial

s 687 – 838 687 The relationship of type of feedback to learning during simulation: a randomized clinical trial Sumit Saraf, Jyothshna Bayya, Howard Minkoff, Jaimie Glick, Nelli Fisher Maimonides Medical Center, OB Gyn, Brooklyn, NY, Maimonides Medical Center, Obstetrics and Gynecology, Brooklyn, NY, Maimonides Medical Center, Ob/Gyn, Brooklyn, NY, Albert Einstein College of Medicine/Montefiore Medical Center, Obstetrics & Gynecology and Women’s Health, Bronx, NY OBJECTIVE: To compare the effect of negative vs. positive feedback on students cognitive and technical performance, and confidence level during obstetrical simulation. STUDY DESIGN: Medical and nursing students without prior experience in performing normal vaginal delivery (NVD) were randomized to positive feedback (PF) or negative feedback (NF) groups. All students (N 59) participated in a teaching demonstration and practiced NVD using a simulator (Noelle 575, Gaumard Scientific). A baseline skills assessment in NVD was followed by a second intervention simulation, during which positive or negative feedback was provided using standardized scripts at predetermined stages of the simulation. Previously validated checklists assessing cognitive and technical performance as well as confidence measures were scored. Mean baseline and intervention scores were compared between groups. For each of the three measures (cognitive, technical, confidence) as dependent variable, a mixed linear model was constructed, with profession (medical, nursing), assessment (baseline, intervention), feedback arm (positive, negative) as factors. RESULTS: Considering both medical and nursing students, all participants in the PF arm demonstrated significant performance improvement after the intervention in all three assessment scores while participants in the NF arm registered a non-significant decline. Specifically, cognitive and technical performance scores, and the confidence scores in the PF group improved significantly with p values of 0.012, 0.026 and 0.036 respectively; while those of the NF group did not (p not statistically significant). CONCLUSION: A positive feedback environment strengthened students cognitive and technical performances and confidence while negative feedback worsened them. Further studies are needed to see if these differences persist over time, and whether they are replicable in a real world environment. 688 Comparison of perinatal outcomes in twins conceived by artificial reproductive techniques and spontaneous conception–a secondary analysis of ESPRiT study data Akila Anbazhagan, Alyson Hunter, Fionnuala Breathnach, Fionnuala McAuliffe, Mike Geary, Sean Daly, John Higgins, John Morrison, Gerry Burke, Shane Higgins, Pat Dicker, Fiona Manning, Steve Carroll, Fergal Malone Royal Jubilee Maternity Hospital, Obstetrics and Gynaecology, Belfast, United Kingdom, Royal College of Surgeons in Ireland, Obstetrics and Gynaecology, Dublin, Ireland, UCD Obstetrics and Gynaecology, School of Medicine and Medical Science, National Maternity Hospital, Holles St, Dublin, Ireland, Rotunda Hospital, Obstetrics and Gynaecology, Dublin, Ireland, The Coombe Women and Infants University Hospital, UCD School of Medicine and Medical Science, UCD, Dublin, Ireland, Cork University Maternity Hospital, Obstetrics, Cork, Ireland, National University of Ireland, Obstetrics & Gynaecology, Galway, Ireland, Regional Maternity Hospital, Obstetrics and Gynaecology, Limerick, Ireland, Our Lady of Lourdes Hospital, Obstetrics and Gynaecology, Drogheda, Ireland, Royal College of Surgeons in Ireland, Epidemiology and Public Health, Dublin, Ireland, National Maternity Hospital, Holles St, Dublin, Ireland OBJECTIVE: To compare perinatal outcomes of twin pregnancies conceived by assisted conception with those of natural conception. STUDY DESIGN: Evaluation of Sonographic Predictors of Restricted growth in Twins (ESPRiT) is a multicentric prospective observational study conducted in eight maternity centres in Ireland, by the Perinatal Ireland Research consortium. Of 1001 twin pregnancies completing the study over a 24 month period, method of conception was available for 874 twins. Statistical comparisons were made between methods of conception and types of artificial reproductive techniques (ARTs) for demographics, baseline characteristics and perinatal outcomes. RESULTS: There were 636 (73%) spontaneously conceived twins and 238 (27%) conceived by ART in the study. Nearly all (97%) of the ART twins were dichorionic as compared to 74% dichorionic twins of those conceived naturally. The ART group was older with a maternal age of 35 years as compared to 31 years for the spontaneous conception group (p 0.0001). The ART group was also more likely to be nulliparous (70% vs 40%, p 0.0001) and were less likely to be smokers (4% versus 15%, p 0.0001). There were no significant differences between the obstetric features such as preeclampsia, gestational diabetes, antepartum haemorrhage or preterm delivery. There were 16(1%) neonatal deaths in the naturally conceived group and 3(0.6%) in the assisted conception group (p 0.124) We found no difference (p-value 0.05) in gestational age at delivery for the ART group Poster Session V www.AJOG.org Saturday, February 11, 2012 • 10:00 am – 12:00 pm • Marsalis Hall, Dallas Hyatt Regency Supplement to JANUARY 2012 American Journal of Obstetrics & Gynecology S307

Intraoperative adverse events can be compensated by technical performance in neonates and infants after cardiac surgery: A prospective study

Our objective was to define the relationship between surgical technical performance score, intraoperative adverse events, and major postoperative adverse events in complex pediatric cardiac repairs. Infants younger than 6 months were prospectively followed up until discharge from the hospital. Technical performance scores were graded as optimal, adequate, or inadequate based on discharge echocardiograms and need for reintervention after initial surgery. Case complexity was determined by Risk Adjustment in Congenital Heart Surgery (RACHS-1) category, and preoperative illness severity was assessed by Pediatric Risk of Mortality (PRISM) III score. Intraoperative adverse events were prospectively monitored. Outcomes were analyzed using nonparametric methods and a logistic regression model. A total of 166 patients (RACHS 4-6 [49%]), neonates [50%]) were observed. Sixty-one (37%) had at least 1 intraoperative adverse event, and 47 (28.3%) had at least 1 major postoperative adverse event. There was no correlation between intraoperative adverse events and RACHS, preoperative PRISM III, technical performance score, or postoperative adverse events on multivariate analysis. For the entire cohort, better technical performance score resulted in lower postoperative adverse events, lower postoperative PRISM, and lower length of stay and ventilation time (P < .001). Patients requiring intraoperative revisions fared as well as patients without, provided the technical score was at least adequate. In neonatal and infant open heart repairs, technical performance score is one of the main predictors of postoperative morbidity. Outcomes are not affected by intraoperative adverse events, including surgical revisions, provided technical performance score is at least adequate.

Year One Outcomes Assessment of a Masters Suturing and Knot-Tying Program for Surgical Interns

We implemented and evaluated the outcomes of a proficiency-based program to enhance basic suturing and knot-tying skills in surgical interns. A suturing and knot-tying "masters program" was implemented for 9 surgical interns. Assessment consisted of timed and videotaped technical performance (TP) of 6 suturing (simple interrupted, subcuticular, and vertical mattress) and tying tasks (1-handed, 2-handed, and tie on pass) preinstruction (baseline) and at 4 months. Feedback and remediation were done after 4 months, and interns were retested until goals were met. TP was scored on a 1-5 scale (1 = novice, 3 = proficient, and 5 = expert). The results were compared with 6 end of PGY2 year residents and rotation evaluations, case numbers, and American Board of Surgery In-Training Examination (ABSITE) scores. The data are mean ± standard deviation (SD); statistical analysis was by 1-way ANOVA, Kruskal-Wallis test, and Pearson correlation. The total combined task times improved significantly from initial (1289 ± 301 seconds) to final assessment (770 ± 139 seconds, p = 0.0003) and between 4-month assessment (1092 ± 253 seconds) and final assessment (p = 0.0237), but not baseline to 4-month assessment (p = 0.213). Final TP scores were improved significantly (baseline, 1.87 ± 0.81; 4-month assessment, 2.63 ± 0.75; and final assessment 3.51 ± 0.39 [p < 0.001]. All interns reached proficiency on 6/6 tasks but not until 10.1 months (range, 4-12 months) and 2.4 assessment sessions (range, 1-3). No significant differences in final total task times or TP scores were found between interns (770 ± 139 seconds and TP, 3.51 ± 0.39) and end of PGY-2 residents (677 ± 109 seconds and TP, 3.6 ± 0.43). The total task time and rotation technical comments were associated significantly, but TP scores, case numbers, and ABSITE scores were not. Training, feedback, and remediation are necessary to reach proficiency in basic suturing and knot tying. This program provides an objective assessment of interns' skills not discernible by conventional measures.

Early Postoperative Severity of Illness Predicts Outcomes After the Stage I Norwood Procedure

We hypothesize that a measure of the immediate postoperative severity of illness after the stage I Norwood operation reflects technical performance or the adequacy of anatomic repair and can serve as a predictor of hospital mortality, reinterventions, and clinical outcomes. One hundred thirty-five patients undergoing stage I were retrospectively studied (2004 to 2007). The severity of illness on postoperative day 1 (POD1) was measured using the Pediatric Risk of Mortality III (PRISM) scoring system. Technical performance scores (optimal, adequate, inadequate) were calculated before hospital discharge. Hospital mortality, postoperative reinterventions, and complications were recorded. Postoperative reintervention was defined as need for cardiac catheterization laboratory or operating room based procedure that included balloon dilation or repair of arch obstruction, shunt revision, reoperations for bleeding, and extracorporeal membrane oxygenation support. Hospital mortality was 14.1% (n=19). The rate of complications and reinterventions was, respectively, 28.1% (n=38) and 26.7% (n=36). The POD1 PRISM score was associated with technical performance (p=0.003). Higher POD1 PRISM scores were associated with mortality (p<0.001), complications (p<0.001), and reinterventions (p=0.001). The POD1 PRISM score had high discrimination for mortality, complications, reinterventions, and inadequate technical performance (areas under the receiver operating characteristic curve were 0.835, 0.776, 0.773, and 0.710, respectively; p≤0.001 for all). The severity of illness as measured by PRISM score on POD1 after the stage I Norwood operation has strong association and discrimination with hospital mortality, postoperative reinterventions, inadequate technical performance, and major postoperative complications. It may be used as an early surrogate of technical performance to initiate a search for and correction of technical deficiencies.

Assessing team performance in the operating room: Development and use of a “black-box” recorder and other tools for the intraoperative environment

The objective of this research was to develop a digital system to archive the complete operative environment along with the assessment tools for analysis of this data, allowing prospective studies of operative performance, intraoperative errors, team performance, and communication. Ability to study this environment will yield new insights, allowing design of systems to avoid preventable errors that contribute to perioperative complications. A multitrack, synchronized, digital audio-visual recording system (RATE tool) was developed to monitor intraoperative performance, including software to synchronize data and allow assignment of independent observational scores. Cases were scored for technical performance, participants' situational awareness (knowledge of critical information), and their comfort and satisfaction with the conduct of the procedure. Laparoscopic cholecystectomy (n = 10) was studied. Technical performance of the RATE tool was excellent. The RATE tool allowed real time, multitrack data collection of all aspects of the operative environment, while permitting digital recording of the objective assessment data in a time synchronized and annotated fashion during the procedure. The mean technical performance score was 73% +/- 28% of maximum (perfect) performance. Situational awareness varied widely among team members, with the attending surgeon typically the only team member having comprehensive knowledge of critical case information. The RATE tool allows prospective analysis of performance measures such as technical judgments, team performance, and communication patterns, offers the opportunity to conduct prospective intraoperative studies of human performance, and allows for postoperative discussion, review, and teaching. This study also suggests that gaps in situational awareness might be an underappreciated source of operative adverse events. Future uses of this system will aid teaching, failure or adverse event analysis, and intervention research.