Accreditation Council For Graduate Medical Education Case Log Research Articles

Representativeness of Workplace-Based Operative Performance Assessments for Resident Operative Experience

Workplace-based assessment is increasingly prevalent in surgical education, especially for assessing operative skill. With current implementations, not all observed clinical performances are assessed, in part because trainees often have discretion about when they seek assessment. As a result, these samples of observed operative performances may not be representative of the full breadth of experience of surgical trainees. Therefore, analyses of these samples may be biased. We aimed to benchmark patterns of procedures logged in the SIMPL operative performance assessment system against records of trainee experience in Accreditation Council for Graduate Medical Education (ACGME) case logs. We analyzed SIMPL longitudinal intraoperative performance assessments from categorical trainees in US general surgery residency programs. We compared overall patterns of how procedures are logged in SIMPL and in ACGME case logs using a Pearson correlation, and we examined differences in how individual procedures are logged in each system using Fisher's exact test. Total procedure frequency from the SIMPL dataset was strongly correlated with total procedure frequency from ACGME case logs (r = 0.86, 95% CI 0.80-0.90). A subset of these procedures (10 of 116 procedures) was logged more frequently in the SIMPL dataset. These 10 procedures accounted for 56% of SIMPL observations and 30% of ACGME logged cases. Case complexity was comparable for assessments initiated by residents and faculty. Samples of intraoperative performance ratings gathered using the SIMPL application largely resemble ACGME case logs. There is no evidence to indicate that residents preferentially select fewer complex cases for assessment.

ACGME Case Log Reminder Does Not Improve Resident Accuracy in Logging Cases.

To assess competency of residents prior to graduation, the Accreditation Council for Graduate Medical Education (ACGME) maintains a case log system, where residents self-report cases they perform. This mechanism results in underreporting of resident involvement in patient care. To determine if an intraoperative case log reminder would increase the frequency of ACGME case logging amongst anesthesiology residents. An intraoperative ACGME case log reminder was implemented on March 13, 2019. The authors collected data for all 53 PGY2-4 anesthesiology residents at the authors' institution from July 14, 2018 to July 16, 2019 from the electronic medical record and ACGME system to calculate the proportion of cases logged and the "lag time" between case occurrence and logging. Data was analyzed for all residents, classes, and individuals. A total of 16,342 anesthetics were performed, and a total of 11,713 cases were logged. The reminder did not improve overall logging rates. Case-logging rates amongst PGY2 residents remained unchanged and declined for PGY3 and PGY4 residents. The lag time between case occurrence and logging increased. An automatic reminder did not improve logging frequency. This may be because residents are unable to log cases intraoperatively in many instances, or they may not feel as though they have participated enough in a case to log it. Additionally, senior residents may log cases less frequently once they have met required case minimums. An automatic case-logging system that transmits resident information directly to the ACGME may be the best way to increase logging accuracy.

ACGME Operative Case Log Accuracy Varies Among Surgical Programs

This study evaluates the accuracy of reported the Accreditation Council for Graduate Medical Education (ACGME) operative case logs from graduated residents compared to institutional operating room electronic records (ORER). We hope this will help guide review committees and institutions develop complete, accurate resident case logs. This is a retrospective, cross-sectional study of general surgery (GS), neurosurgery (NS), and orthopedic surgery (OS) resident physicians. ACGME and ORER cases from 2009 to 2010 were analyzed and each case and current procedural terminology (CPT) code directly compared (ORER vs. ACGME). Single academic tertiary-care medical center (University of Kentucky, Lexington, KY). Eleven thousand nine hundred and twenty-three cases for 46 residents among the 3 residency programs were analyzed. There was an overall logging accuracy of 72% for ORER cases reflected in the ACGME case logs. OS residents had a higher rate of logging accuracy (OS 91%, GS 69%, NS 58%, chi-square p = 0.014) and mean annual number of cases compared to the other 2 programs (OS 452, GS 183, NS 237, ANOVA p = 0.001). NS residents had higher accuracy of CPT codes than post-graduate years 2 to 5 in other programs (p < 0.017). There was a strong positive correlation between the number of cases completed per resident and case logging accuracy, (rho = 0.769, p < 0.001) consistent for NS and GS, but not OS. This study shows only 72% of a residents' operative experience is captured in the ACGME case log across 3 surgical programs. There is significant variability among surgical programs and among post-graduate year cohorts regarding case log and CPT code accuracy. There is a strong correlation with the total number of cases performed and increasing case log accuracy. Low case log accuracy may reflect individual resident behavior instead of program operative exposure. Further studies are needed to determine if ORER may serve as a more complete assessment of the operative experience of a resident and program.

Lessons Learned in Maximizing Open Surgical Experience for Vascular Integrated Residents During the Nonvascular Months of Training

Open surgical experience remains a highly coveted area of vascular integrated surgery (VIS) training. VIS training recently decreased the number of required nonvascular rotations (NVR) from 24 to 18 months with further decreases on the horizon. Cooper University Hospital VIS program has developed a curriculum that maximizes open surgical exposure during NVR. The purpose of this study is to identify and compare our experience with national trends, identifying site specific and national high yield open experience NVRs. The Accreditation Council for Graduate Medical Education (ACGME) case log graduation statistics between 2012 and 2018 were compiled. Our institution’s ACGME case logs from 2015 to 2019 were likewise compiled. Primary outcome was the average number of nonvascular cases performed as first assist, surgeon junior, surgeon chief and teach assist. Secondary outcomes were the number of open vs minimally invasive cases (laparoscopic, thoracoscopic, etc) by VIS residents in abdominal, alimentary tract, trauma, thoracic, plastic surgery and transplant categories. A review of the VIS rotation schedules from the ACGME accredited programs was also performed. Number and type of NVR rotations were calculated by postgraduate year (PGY). A total of 185 VIS residents completed training during the study period. The abdominal category averaged the greatest number of cases during NVRs overall, however also had the greatest percentage of minimally invasive (laparoscopic) cases. Transplant and plastic surgery had the highest percentage of open cases (100%) (Fig 1). Our institution’s residents had comparable open and laparoscopic experience except in the cardiothoracic category, where the number of cases performed far exceeded the national average (Fig 2). When evaluating rotation schedules, 28 VIS programs’ schedules were available for review (46.7%). The top three PGY 1 rotations were general surgery followed by trauma and plastic surgery. The top three PGY 2 rotations were general surgery followed by critical care and transplant. Much of today’s general surgery experience for VIS residents on NVRs are minimally invasive surgeries. Both nationally and at Cooper, transplant and trauma provide the highest percentage of open surgical experience. Cardiothoracic cases provide the highest number of open cases both at our institution and nationally. General surgery is the most common NVR among VIS programs even though it provides a lower yield of open surgery experience for VIS residents. Increasing the number of cardiac surgery months in place of low-yield rotations may increase open surgical training for VIS residents.Fig 2Surgical experience nationally and at our institution on nonvascular rotations stratified by open vs minimally invasive.View Large Image Figure ViewerDownload Hi-res image Download (PPT)

Case Reporting, Competence, and Confidence: A Discrepancy in the Numbers

The Accreditation Council for Graduate Medical Education (ACGME) continues to play an integral role in accreditation of surgical programs. The institution of case logs to demonstrate competency of graduating residents is a key component of evaluation. This study compared the number of vascular cases a surgical resident has completed according to the ACGME operative log to their operative proficiency, quality of anastomosis, operative experience, and confidence in both a simulation and operative setting. General surgery residents ranging from PGY 1 to 5 participated in a simulation laboratory in which they completed an end-to-side vascular anastomosis. Each participant was given a weighted score based on technical proficiency and anastomosis quality using a previously validated Global Rating Scale (Duran et al, 2014). These scores were correlated to the General Surgery Milestones. Participants completed preoperative and postoperative surveys assessing resident operative experience using the 4-level Zwisch scale (DaRosa et al., 2013), confidence with vascular procedures and confidence performing simulated anastomoses. Confidence was assessed on a scale from 1 to 9 (not confident to extremely confident). Case logs were recorded for each participant. An IRB approved questionnaire was distributed to assess preoperative and postoperative roles of both the resident physician and faculty, with a defined goal. Univariate and multivariate analysis was performed. Twenty-one general surgery residents were evaluated in the simulation laboratory and 8 residents were assessed intraoperatively. The residents were evenly distributed throughout clinical years. Groups of residents were divided into quartiles based upon the number of vascular cases recorded in the ACGME database. No correlation was found between number of cases, Milestones score and the weighted score (p = 0.94). No statistical significance was found between confidence and quality of anastomosis (p = 0.1). Resident operative experience per the Zwisch scale was categorized most commonly as "Smart Help" by both the trainee and attending surgeon, despite mean resident confidence ratings of 6.67 (± 1.61) with vascular procedures. ACGME case logs, which are utilized to assess readiness for completion of general surgery residency, may not be indicative of a resident's operative competency and technical proficiency. Confidence is not correlated with technical ability. Faculty and resident insight as to their role in a procedure differ, as faculty feel that they are providing less help than the resident perceives. Careful examination of resident operative technique is the best measure of competency.

Variation in National ACGME Case Log Data for Pediatric Orthopaedic Fellowships: Are Fellow Coding Practices Responsible?

The introduction of the 80-hour work week for Accreditation Council for Graduate Medical Education (ACGME) accredited fellowship programs initiated many efforts to optimize surgical training. One particular area of interest is on recording and tracking surgical experiences. The current standard is logging cases based on Current Procedural Terminology codes, which are primarily designed for billing. Proposed guidelines from the ACGME regarding logging exist, but their implementation is unknown, as is the variation in case volume across fellowship programs. The purpose of this study was to investigate variability in the national case log data, and explore potential sources of variation using fellow surveys. National ACGME case log data for pediatric orthopaedic fellowships from 2012 to 2015 were reviewed, with particular attention to the domains of spine, pelvis/hip, arthroscopy, trauma, and other (which includes clubfoot casting). To explore potential sources of case log variability, a survey on case logging behavior was distributed to all pediatric orthopaedic fellows for the academic year 2015 to 2016. Reported experiences based on ACGME case logs varied widely between fellows with percentage difference of up to 100% in all areas. Similarly, wide variability is present in coding practices of pediatric orthopaedic fellows, who often lack formal education on the topic of appropriate coding/logging. In the survey, hypothetical case scenarios had an absolute difference in recorded codes of up to 13 and a percentage difference of up to 100%. ACGME case log data for pediatric orthopaedic fellowships demonstrates wide variability in reported surgical experiences. This variability may be due, in part, to differences in logging practices by individual fellows. This observation makes meaningful interpretation of national data on surgical volume challenging. Proposed surgical experience minimums should be interpreted in light of these data, and may not be advisable unless accompanied by standardized and specific guidelines for case log entry. Efforts to optimize training in the post 80-hour era will require accurate data to serve as a starting point for future educational efforts.

Assessing the accuracy of neurological surgery resident case logs at a single institution

Background:Despite the importance of case logs in evaluating residents, no studies assess their accuracy in neurological surgery. Studies from other specialties reveal variations in reporting. This study assesses the accuracy of neurological surgery resident case logs at a single institution.Methods:Data was collected from three databases: billing data and two separate resident-managed case logs [department log and Accreditation Council for Graduate Medical Education (ACGME) case logs], containing records of procedures performed by 14 neurological surgery residents at a single institution over a 1-year period. The billing data was used as a proxy for a census of procedures performed during the study period. The difference between the number of procedures logged by residents and the number of procedures billed was calculated to determine the accuracy of the resident case logs.Results:Over the study period, 2150 procedures were billed at the institution, whereas 1749 procedures were logged in the ACGME case log and 1873 in the department log, representing an error rate of −18.65% and −12.88%, respectively. The error rate varied significantly (−1150% to +50.23%) between ACGME procedure categories. In 13 of the 22 ACGME procedure categories, the procedures were under-logged by residents in both resident-managed case logs. No category demonstrated over-logging in both case log systems.Conclusion:Resident managed case logs are an incomplete representation of clinical work. The cause for inaccuracy is multifactorial. The authors suggested that further research is necessary to validate their results and to identify means by which the accuracy of case logs can be increased.

The Design and Implementation of an Automated System for Logging Clinical Experiences Using an Anesthesia Information Management System

Residents in anesthesia training programs throughout the world are required to document their clinical cases to help ensure that they receive adequate training. Current systems involve self-reporting, are subject to delayed updates and misreported data, and do not provide a practicable method of validation. Anesthesia information management systems (AIMS) are being used increasingly in training programs and are a logical source for verifiable documentation. We hypothesized that case logs generated automatically from an AIMS would be sufficiently accurate to replace the current manual process. We based our analysis on the data reporting requirements of the American College of Graduate Medical Education (ACGME). We conducted a systematic review of ACGME requirements and our AIMS record, and made modifications after identifying data element and attribution issues. We studied 2 methods (parsing of free text procedure descriptions and CPT4 procedure code mapping) to automatically determine ACGME case categories and generated AIMS-based case logs and compared these to assignments made by manual inspection of the anesthesia records. We also assessed under- and overreporting of cases entered manually by our residents into the ACGME website. The parsing and mapping methods assigned cases to a majority of the ACGME categories with accuracies of 95% and 97%, respectively, as compared with determinations made by 2 residents and 1 attending who manually reviewed all procedure descriptions. Comparison of AIMS-based case logs with reports from the ACGME Resident Case Log System website showed that >50% of residents either underreported or overreported their total case counts by at least 5%. The AIMS database is a source of contemporaneous documentation of resident experience that can be queried to generate valid, verifiable case logs. The extent of AIMS adoption by academic anesthesia departments should encourage accreditation organizations to support uploading of AIMS-based case log files to improve accuracy and to decrease the clerical burden on anesthesia residents.

Is Surgical Resident Comfort Level Associated With Experience?

The Accreditation Council for Graduate Medical Education (ACGME) has placed great emphasis on residents learning to identify their training needs and to develop learning strategies to address these needs. In surgery, residents can play an active role in identifying training needs through self-assessment of their procedural skills. Our study contributes to the growing body of literature regarding practice-based learning and improvement by attempting to determine if surgery resident experience is associated with comfort level and perceived training needs. Twenty-five general surgery residents completed a surgical experience survey, which asked them to indicate the range of times they performed a procedure, their comfort level in performing the procedure, and the quality of training they felt they received. Residents were given a survey with cases appropriate for their postgraduate year. A Cochran-Armitage trend test was used to evaluate the trends between comfort level and experience, and training needs and experience. A P value of 0.05 was considered statistical significance in all analyses. Resident comfort level demonstrated a positive trend compared with case volume both in the self-reported survey and ACGME case log (P values<0.001). Additionally, higher levels of training were associated with increased comfort level (P value=0.05). Perceived training needs and experience were also associated (P value<0.001), demonstrating that with increased experience, residents felt that their training needs were being met. Our study demonstrates that residents are able to assess their comfort level and training needs based on both actual and perceived experience. The procedural survey has been a useful tool for resident self-assessment in that residents are able to play a more active role in their education by developing appropriate learning plans.