Blinded peer assessment of surgical skill is feasible and can predict complication rates: a step toward measuring surgical quality.

Abstract

Despite a growing emphasis on measuring and grading quality of surgical care, a tremendous knowledge gap exists regarding what constitutes quality care. How to measure quality of care remains equally daunting in scope. Thus far, the focus has been on perioperative metrics that are relatively easy to measure and monitor such as deep venous thrombosis prophylaxis. It is doubtful that these perioperative factors will be dominant determinants of operative outcomes. While measuring intraoperative factors seems rational on the surface, huge complexities arise when one looks beyond the surface. Operative time, for example, can be a misleading measure of quality and may or may not relate directly to ultimate outcome. Even though volume has been grossly linked to outcomes, individual surgeon skills have been harder to define, study, and monitor. Which factors can affect surgeon skills and how these skills may relate to complications remains largely speculative. To investigate this important issue, Birkmeyer et al conducted a study to objectively assess the surgical performance of 20 bariatric surgeons located in Michigan, and to examine the relationship between their surgical skill and postoperative outcomes. The assessment of surgical skill was performed by at least 10 blinded peer surgeons, and the risk-adjusted complication rates were studied using a prospectively collected outcome registry. The results were published in the October 2013 issue of The New England Journal of Medicine.1 Twenty of 63 bariatric surgeons in Michigan who perform laparoscopic gastric bypass surgeries volunteered to participate in the study. Between August 2006 and 2012, the 20 participating surgeons performed bypass procedures on 10 343 patients who were prospectively enrolled in the Michigan Bariatric Surgery Collaborative (MBSC) registry.2,3 The authors used this registry to study postoperative complications and to correlate them with surgical performance. Initially, to assess the surgical skill of the participating surgeons, each surgeon was asked to submit an operative video of his/her choice. These videos were edited to include only important elements of the procedure (range of video time was 25-40 minutes). Each video was rated by at least 10 independent peer surgeons who were blinded to both the operator and the surgical outcome of the procedure. The authors used the average grading of all peer surgeons for each video. The grading was performed using a 5-point, likert-type scale derived from the validated Objective Structured Assessment of Technical Skills (OSATS).4-6 The scale involved five main items: instrument handling, tissue exposure, time and motion, gentleness, and flow of operation, in addition to an overall skill rating score. Using the MBSC database, the authors studied the surgical outcomes of procedures performed by each of the participating surgeons. The primary outcome measures included postoperative complications such as wound infection or dehiscence, abdominal abscess, leak, anastomotic stricture, bowel obstruction, and/or bleeding. Other medical complications including venous thromboembolism, pneumonia, respiratory failure, renal failure, myocardial infarction, and death were also studied. Additionally, the authors reported on the 30-day mortality, readmission and reoperation rates, as well as emergency department visits. The data analysis involved the surgical and medical outcomes of patients treated by each surgeon. Surgeon-specific ratings were used to assess the relationship between surgical skill and outcomes. To facilitate data presentation, the surgeons were classified into four groups or quartiles based on their surgical skill ratings. For each quartile, a risk-adjusted complication rate was calculated. The surgeons were later categorized into a low performance group (1st quartile), an intermediate performance group (2nd and 3rd quartiles), and a high performance group (4th quartile). The results of surgical skill rating showed a wide range of skill among the participating surgeons. Out of a maximum score of 5, the average score of surgeons in the low performance group (n = 5) was 2.9 compared to 4.4 in the high performance group (n = 5) (Table). Surgical skill was found to significantly correlate with the average number of laparoscopic gastric bypass procedures performed per year (53 vs 157 procedures for the low vs high performance groups, P = .005). Similarly, surgeons in the high performance group demonstrated significantly shorter operating times compared to surgeons in the low performance group (98 vs 137 minutes, respectively, P < .001) (Table). Interestingly, other factors such as the duration of practice, completion of a fellowship, or practicing at a teaching hospital did not correlate with surgical skill. No significant difference was noted among patient populations of various groups. Surgical skill was found to have a linear correlation with postoperative complication rates of individual surgeons (Figure). The low performance group was noted to have a higher risk-adjusted complication rate compared to the high performance group (14.5% vs 5.2, respectively, P < .001). Interestingly, the surgical skill scores correlated well with both surgical and medical complications. The rates of infection, pulmonary complications, and bowel obstruction were significantly lower in the high performance group. Surgeons in the low performance group demonstrated a mortality rate of 0.26% compared to 0.05% in the high performance group (P = .01). Similarly, compared to the high performance group, the low performance group demonstrated significantly higher rates of reoperation (3.4% vs 1.6%, P = .01), readmissions (6.3% vs 2.7%, P < .001), and emergency room visits within 30 days (10.2% vs 3.8%, P = .004).Table: Peer Rating of Surgical Skill and its Relationship With Type of Surgery, Patient Volume, Procedure Volume, Mean Operating Room Time, Completion of Fellowship and Practice at a Teaching HospitalFigure: The relationship between risk-adjusted complication rates and peer surgeon rating of surgical skill. Each diamond in the scatter plot represents 1 of 20 practicing bariatric surgeons. From [New England Journal of Medicine, Birkmeyer JD, Finks JF, O'Reilly A, et al., Surgical skill and complication rates after bariatric surgery, Volume No. 369, Page No. 1434-1442. Copyright © (2013). Massachusetts Medical Society. Reprinted with permission from Massachusetts Medical Society.There are several inherent limitations in the current study. These include the fact that only 30% of surgeons volunteered to participate and the fact that each surgeon submitted a “representative case” of his/her choice. Nevertheless, the surgeons in Michigan are to be congratulated for creating this registry and for being brave enough to submit videos for rating. Furthermore, the authors are to be congratulated for publishing these compelling results, which support the common belief that surgical skill is a potent determinant of outcome. The results of this study reinforce the inverse relationship between surgical volume and complications, and thus shine light on the importance of regionalization of care. However, in neurosurgery, standardization of care through guidelines and national registries needs to be accomplished prior to regionalizing care. What is most intriguing about this study, though, is the potential of stratifying skill based on an objective blinded peer review using a simple scale. This opens up many fascinating opportunities to improve outcomes. Whether this approach will work in neurosurgery remains to be determined, but certainly merits rigorous investigation. It is important to note that this study would not have been possible without the high quality registry which was in place to rigorously monitor outcomes of bariatric surgery in Michigan. While objective feedback is known to be an important element on the road to skills mastery,7-9 there is no good mechanism to do this, particularly after residency training. Since volume appears to be a strong determinant of skill, it is possible that simulation can be used to enhance skills for those surgeons who are identified to be lagging in skill. If we are to achieve the goal of improving surgeon skills, an objective way to measure, monitor, and inform surgeons about their skills is desperately needed. This study is one step in that direction.