System For Improving And Measuring Procedural Learning Research Articles

Keep it SIMPL: Improved Feedback After Implementation of an App-Based Feedback Tool.

Data defining the utility of the system for improving and measuring procedural learning (SIMPL) in surgical education is limited. The aim of this pilot study is to describe the impact of SIMPL on resident and faculty perspectives regarding operative feedback. Residents and faculty were surveyed prior to and 6months after SIMPL implementation. Associations were analyzed using for categorical and Student's t-test for continuous variables. Statistical significance was defined as P-value < .05. The proportion of residents receiving intraoperative feedback at least once/day increased significantly (35% to 73%, P = .025); there was a trend toward increased postoperative feedback (15% to 33%, P = .201). Faculty reported an increase in intraoperative (55% to 91%, P = .041) and postoperative feedback (21% to 64%, P = .020). Satisfaction with intraoperative feedback improved from a score of 3.50 ± 1.05 to 3.93 ± .62, although not statistically significant (P = .181). Satisfaction with postoperative feedback improved significantly from 2.85 ± .93 to 3.50 ± .65 (P = .032). The proportion of faculty who felt they were providing effective feedback increased (53% to 91%, P = .032). The proportion of residents who perceived that feedback enhanced their surgical skill increased as well (65% to 93%, P = .048). Despite a modest increase in frequency of postoperative feedback, perceived quality of feedback improved substantially after implementation of SIMPL. Introduction of SIMPL also increased the amount of feedback provided by faculty intraoperatively. SIMPL, via direct and indirect effects, has a positive impact on the resident operative learning environment. Further work is necessary to examine the influence this may have on resident operative skill and patient outcomes.

Are You Thinking What I'm Thinking? Exploring Response Process Validity Evidence for a Workplace-based Assessment for Operative Feedback

Workplace-based assessments (WBAs) are used in multiple surgical specialties to facilitate feedback to residents as a form of formative assessment. The validity evidence to support this purpose is limited and has yet to include investigations of how users interpret the assessment and make rating decisions (response processes). This study aimed to explore the validity evidence based on response processes for a WBA in surgery. Semi-structured interviews explored the reasonings and strategies used when answering questions in a surgical WBA, the System for Improving and Measuring Procedural Learning (SIMPL). Interview questions probed the interpretation of the three assessment questions and their respective answer categories (level of autonomy, operative performance, case complexity). Researchers analyzed transcripts using directed qualitative content analysis to generate themes. Single tertiary academic medical center. Eight residents and 13 faculty within the Department of Otolaryngology-Head and Neck Surgery participating in a 6-month pilot of SIMPL. We identified four overarching themes that that characterized faculty and resident response processes while completing SIMPL: (1) Faculty and resident users had similar content-level interpretations of the questions and corresponding answer choices; (2) Users employed a variety of cognitive, behavioral, and emotional processes to make rating decisions; (3) Contextual factors influenced ratings; and (4) Tensions during interpretation contributed to rating uncertainty. Response processes are a key source of evidence to support the validity for the formative use of WBAs. Evaluating response process evidence should go beyond basic content-level analysis as contextual factors and tensions that arise during interpretation also play a large role in rating decisions. Additional work and a continued critical lens are needed to ensure that WBAs can truly meet the needs for formative assessment.

Smartphone-based app for evaluating cardiothoracic residents: Feasibility and engagement.

Timely and high-quality feedback is important in cardiothoracic (CT) surgery education. Feedback on operative proficiency is an area for improvement in CT surgery programs. Traditional evaluations significantly lag behind operative interactions. We hypothesized that use of the System for Improving and Measuring Procedural Learning (SIMPL) app would improve operative feedback for trainees. Use of SIMPL was evaluated from December 2018 to January, 2021 within an academic CT surgery training program. Ratings include level of supervision, complexity of the operation, and trainee performance. Completion was limited to 72 h after the operation. Descriptive statistics of the users and ratings are presented. Over 28 months, 816 evaluations were completed, and of these, 495 had a rating from both the faculty and trainee. There were 19 trainees representing post-graduate years 1-8 and 19 faculty members who received or submitted at least one evaluation over the study period. The number of evaluations for each trainee ranged from 1 to 166 and from 1 to 81 for each of the faculty. The response rate for faculty ranged from 0% to 100%. "Active help" was the most common type of supervision (50.7% by the faculty, 60.4% from the trainees). Use of SIMPL within a CT surgery training program was feasible and engagement was observed from both trainees and faculty. SIMPL provided trainees with timely, concise feedback on operative performance. Further work will focus on correlating SIMPL ratings with pre-existing assessments of performance.

Correlation of Supervised Independence and Performance with Procedure Difficulty amongst Surgical Residents Stratified by Post Graduate Year

This study investigates the role of procedure difficulty on attending ratings of supervised levels of independence and procedural performance amongst general surgery residents, while accounting for case complexity. Attending ratings for residents were obtained from System for Improving and Measuring Procedural Learning (SIMPL) database. Current procedural terminology (CPT) codes were used to match procedures to a corresponding work relative value unit (wRVU) as a surrogate for procedure difficulty. Three categories of wRVU (<13.07, 13.07-22, >22) were identified using recursive partitioning. Procedures were also divided into 'Core' or 'Advanced' as defined by the American Board of Surgery Surgical Council on Resident Education (SCORE). Temporal advancement in resident skill was accounted for through academic quarterly analysis. A generalized estimating equations (GEE) approach was used to form separate multivariable logistic regression models for meaningful autonomy (MA) and satisfactory performance (SP) adjusted for potential clustering by program, subject, and rater. Models were further adjusted for core/advanced procedures, attending rated complexity, and academic quarter. A total of 33,281 ratings were analyzed. Overall, 51.6% were rated as MA and 44.4% as SP. For core procedures, surgical residents rated as MA (53.5%) and SP (45.7%), which was twice as high as those for advance procedures (MA-29.2%, SP-29.0%). MA and SP both decreased with increasing wRVU (Figure 2 &3). Using a wRVU<13.07 as a reference, the adjusted odds ratios of MA and SP were significantly lower with increasing procedure difficulty, 0.44 for wRVU 13.07-22.0 and 0.24 for wRVU >22.00 (Table 3). Post graduate year (PGY) 5 residents in the final quarter of training obtain MA in 95.5% and SP 92.9% for core procedures with wRVU <13.07 (Table 4). Increasing procedural difficulty is independently associated with decreases in meaningful autonomy and satisfactory performance. As residents approach graduation the level of meaningful autonomy and satisfactory performance both reach high levels for common core procedures but decrease as procedural difficulty increases.

The Quality of Operative Performance Narrative Feedback: A Retrospective Data Comparison Between End of Rotation Evaluations and Workplace-based Assessments.

To describe the quality of operative performance feedback using evaluation tools commonly used by general surgery residency training programs. The majority of surgical training programs administer an evaluation through which faculty members may rate and comment on trainee operative performance at the end of the rotation (EOR). Many programs have also implemented the system for improving and measuring procedural learning (SIMPL), a workplace-based assessment tool with which faculty can rate and comment on a trainee's operative performance immediately after a case. It is unknown how the quality of narrative operative performance feedback delivered with these tools compares. The authors collected EOR evaluations and SIMPL narrative comments on trainees' operative performance from 3 university-based surgery training programs during the 2016-2017 academic year. Two surgeon raters categorized comments relating to operative skills as being specific or general and as encouraging and/or corrective. Comments were then classified as effective, mediocre, ineffective, or irrelevant. The frequencies with which comments were rated as effective were compared using Chi-square analysis. The authors analyzed a total of 600 comments. 10.7% of EOR and 58.3% of SIMPL operative performance evaluation comments were deemed effective (P < 0.0001). Evaluators give significantly higher quality operative performance feedback when using workplace-based assessment tools rather than EOR evaluations.

Concordance Between Expert and Nonexpert Ratings of Condensed Video-Based Trainee Operative Performance Assessment

We examined the impact of video editing and rater expertise in surgical resident evaluation on operative performance ratings of surgical trainees. Randomized independent review of intraoperative video. Operative video was captured at a single, tertiary hospital in Boston, MA. Six common general surgery procedures were video recorded of 6 attending-trainee dyads. Full-length and condensed versions (n = 12 videos) were then reviewed by 13 independent surgeon raters (5 evaluation experts, 8 nonexperts) using a crossed design. Trainee performance was rated using the Operative Performance Rating Scale, System for Improving and Measuring Procedural Learning (SIMPL) Performance scale, the Zwisch scale, and ten Cate scale. These ratings were then standardized before being compared using Bayesian mixed models with raters and videos treated as random effects. Editing had no effect on the Operative Performance Rating Scale Overall Performance (-0.10, p = 0.30), SIMPL Performance (0.13, p = 0.71), Zwisch (-0.12, p = 0.27), and ten Cate scale (-0.13, p = 0.29). Additionally, rater expertise (evaluation expert vs. nonexpert) had no effect on the same scales (-0.16 (p = 0.32), 0.18 (p = 0.74), 0.25 (p = 0.81), and 0.25 (p = 0.17). There is little difference in operative performance assessment scores when raters use condensed videos or when raters who are not experts in surgical resident evaluation are used. Future validation studies of operative performance assessment scales may be facilitated by using nonexpert surgeon raters viewing videos condensed using a standardized protocol.

Using a Mobile Application for Evaluation of Procedural Learning in Neurosurgery

Feedback is a major component of any educational process. Competency assessment and feedback are essential tools for teaching surgical skills. Most current systems of assessment within neurosurgical residency programs use evaluations aggregated over a period of time, which lead to significant bias, rather than case-specific feedback. We describe the use of a mobile device application called SIMPL (System for Improving and Measuring Procedural Learning) to allow for immediate surgical competency assessment and improve the quality of feedback after every operative experience. We retrospectively assessed our use of this program and neurosurgery residents' response to the program. Institutional review board approval was not required, and no patients were involved in the study. This application has already been implemented within general surgery programs with promising results. We document its implementation in a neurosurgery program and the output of the program for residents and program directors. This is the first documentation of such an application within a neurosurgical residency program. In a 6-month period at a single institution, around 300 evaluations were completed by residents and faculty, with a >80% response rate within both groups. Furthermore, these evaluations were companied by a 60% dictation rate from faculty, which contains verbal feedback that is available to residents for playback at any time. The telephone-based SIMPL application allows for assessment of surgical competency and remains quick and easy to use, giving it the potential to improve individual neurosurgical training experiences across the United States.

Assessment of Operative Autonomy and Readiness for Independent Practice Among Pediatric Surgery Fellows

PurposeThe purpose of this study was to evaluate the implementation of the System for Improving and Measuring Procedural Learning (SIMPL) “app” in a pediatric surgery training program. SIMPL is a novel workplace-based operative performance assessment platform which allows case by case assessment of trainees using three scales (autonomy, performance, and case complexity) and takes seconds to complete. MethodsA pediatric surgery-specific procedural taxonomy for SIMPL was developed from the ACGME procedural library. SIMPL was piloted and implemented in a single pediatric surgery training program. A descriptive analysis of the operative assessments was undertaken. ResultsIn the year following SIMPL implementation, 565 (39% of cases) assessments were completed by 21 faculty for 3 fellows for 148 unique procedures within a median of 8.5 h. Recorded feedback accompanied 61% of assessments. 2nd year fellows were more likely to be deemed autonomous (81% ratings) and practice ready (84%) vs. 1st year fellows (50%; p < 0.001 both), with improvements over time. ConclusionWithin pediatric surgery, this is the first implementation of a workplace-based operative performance assessment “app”. With its ease of use, SIMPL drastically increased the volume of operative evaluations and diversified the mix of cases evaluated while SIMPL ratings demonstrated expected and graduated performance improvements over time. Type of studyEducation. Level of evidenceN/A.

Using smartphones for trainee performance assessment: A SIMPL case study

Workplace-based assessments are used by raters to evaluate observed performance of trainees in actual clinical practice. These types of assessments are of growing interest, especially because observed performance is prioritized within the larger competency-based medical educational movement. Implementation of workplace-based assessments has, however, been challenging. This article describes the motivations and implications for workplace-based assessments that leverage smartphone technology. It does so in reference to an app called SIMPL (System for Improving and Measuring Procedural Learning) in order to highlight some of the challenges and benefits one might encounter during implementation of similar systems.

Assessments of Otolaryngology Resident Operative Experiences Using Mobile Technology: A Pilot Study.

Surgical education has shifted from the Halstedian model of "see one, do one, teach one" to a competency-based model of training. Otolaryngology residency programs can benefit from a fast and simple system to assess residents' surgical skills. In this quality initiative, we hypothesize that a novel smartphone application called System for Improving and Measuring Procedural Learning (SIMPL) could be applied in an otolaryngology residency to facilitate the assessment of resident operative experiences. The Plan Do Study Act method of quality improvement was used. After researching tools of surgical assessment and trialing SIMPL in a resident-attending pair, we piloted SIMPL across an otolaryngology residency program. Faculty and residents were trained to use SIMPL to rate resident operative performance and autonomy with a previously validated Zwisch Scale. Residents (n = 23) and faculty (n = 17) were trained to use SIMPL using a standardized curriculum. A total of 833 assessments were completed from December 1, 2017, to June 30, 2018. Attendings completed a median 20 assessments, and residents completed a median 14 self-assessments. All evaluations were resident initiated, and attendings had a 78% median response rate. Evaluations took residents a median 22 seconds to complete; 126 unique procedures were logged, representing all 14 key indicator cases for otolaryngology. This is the first residency-wide application of a mobile platform to track the operative experiences of otolaryngology residents. We adapted and implemented a novel assessment tool in a large otolaryngology program. Future multicenter studies will benchmark resident operative experiences nationwide.

Value and Barriers to Use of the SIMPL Tool for Resident Feedback

The System for Improving and Measuring Procedural Learning (SIMPL) is a smart-phone application used to provide residents with an evaluation of operative autonomy and feedback. This study investigated the perceived benefits and barriers to app use. A database of previously performed SIMPL evaluations was analyzed to identify high, low, and never users. Potential predisposing factors to use were explored. A survey investigating key areas of value and barriers to use for the SIMPL application was sent to resident and faculty users. Respondents were asked to self-identify how often they used the app. The perceived benefits and barriers were correlated with the level of usage. Qualitative analysis of free text responses was used to determine strategies to increase usage. General surgery training programs who are members of the Procedural Learning and Safety Collaborative. Surgical residents and faculty. At least 1 SIMPL evaluation was created for 411 residents and 524 faculty. Thirty percent of both faculty and residents were high-frequency users. Thirty percent of faculty were never users. One hundred eighty-eight residents and 207 faculty (response rate 46%) completed the survey. High-frequency resident users were more likely to perceive a benefit for both numerical evaluations (76% vs 30%) and dictated feedback (92% vs 30%). Faculty and residents commonly blamed each other for not creating and completing evaluations regularly (87% of residents, 81% of faculty). Suggested strategies to increase usage included reminders and integration with existing data systems. Frequent users perceive value from the application, particularly from dictated feedback and see a positive impact on feedback in their programs. Faculty engagement represents a major barrier to adoption. Mechanisms which automatically remind residents to initiate an evaluation will help improve utilization but programs must work to enhance faculty willingness to respond and dictate feedback.

Examining the Impact of Using the SIMPL Application on Feedback in Surgical Education

The System for Improving and Measuring Procedural Learning (SIMPL) smartphone application allows physicians to provide dictated feedback to surgical residents. The impact of this novel feedback medium on the quality of feedback is unknown. Our objective was to compare the delivery and quality of best-case operative performance feedback given via SIMPL to feedback given in-person. We collected operative performance feedback given both in-person and via SIMPL from surgeons to residents over 6 weeks. Feedback transcripts were coded using Verbal Response Modes speech acts taxonomy to compare the delivery of feedback. We evaluated quality of feedback using a validated resident survey and third-party assessment form. University of Wisconsin School of Medicine and Public Health, a large academic medical institution. Four surgical attendings and 9 general surgery residents. Nineteen SIMPL and 18 in-person feedback encounters were evaluated. Feedback via SIMPL was more directive (containing thoughts, perceptions, evaluations of resident behavior, or advice) and contained more presumptuous utterances (in which the physician reflected on and assessed resident performance or offered suggestions for improvement) than in-person feedback (p = 0.01). The resident survey showed no significant difference between the quality of feedback given via SIMPL and in-person (p = 0.07). The mean score was 47.74 (SD = 3.00) for SIMPL feedback and 45.33 (SD = 4.77) for in-person feedback, with a total possible score of 50. Third-party assessment showed no significant difference between the quality of feedback given via SIMPL and in-person (p = 0.486). The mean score was 23.40 (SD = 3.75) for SIMPL feedback and 22.25 (SD = 5.94) for in-person feedback, with a total possible score of 30. Although feedback given via SIMPL was more direct and based on the attendings' perspectives, the quality of the feedback did not differ significantly. Use of the dictation feature of SIMPL to deliver resident operative performance feedback is a reasonable alternative to in-person feedback.

Readiness of US General Surgery Residents for Independent Practice.

This study evaluates the current state of the General Surgery (GS) residency training model by investigating resident operative performance and autonomy. The American Board of Surgery has designated 132 procedures as being "Core" to the practice of GS. GS residents are expected to be able to safely and independently perform those procedures by the time they graduate. There is growing concern that not all residents achieve that standard. Lack of operative autonomy may play a role. Attendings in 14 General Surgery programs were trained to use a) the 5-level System for Improving and Measuring Procedural Learning (SIMPL) Performance scale to assess resident readiness for independent practice and b) the 4-level Zwisch scale to assess the level of guidance (ie, autonomy) they provided to residents during specific procedures. Ratings were collected immediately after cases that involved a categorical GS resident. Data were analyzed using descriptive statistics and supplemented with Bayesian ordinal model-based estimation. A total of 444 attending surgeons rated 536 categorical residents after 10,130 procedures. Performance: from the first to the last year of training, the proportion of Performance ratings for Core procedures (n = 6931) at "Practice Ready" or above increased from 12.3% to 77.1%. The predicted probability that a typical trainee would be rated as Competent after performing an average Core procedure on an average complexity patient during the last week of residency training is 90.5% (95% CI: 85.7%-94%). This falls to 84.6% for more complex patients and to less than 80% for more difficult Core procedures. Autonomy: for all procedures, the proportion of Zwisch ratings indicating meaningful autonomy ("Passive Help" or "Supervision Only") increased from 15.1% to 65.7% from the first to the last year of training. For the Core procedures performed by residents in their final 6 months of training (cholecystectomy, inguinal/femoral hernia repair, appendectomy, ventral hernia repair, and partial colectomy), the proportion of Zwisch ratings (n = 357) indicating near-independence ("Supervision Only") was 33.3%. US General Surgery residents are not universally ready to independently perform Core procedures by the time they complete residency training. Progressive resident autonomy is also limited. It is unknown if the amount of autonomy residents do achieve is sufficient to ensure readiness for the entire spectrum of independent practice.

Evaluation of Oral and Maxillofacial Surgery Residents' Operative Skills: Feasibility and Engagement Study Using SIMPL Software for a Mobile Phone

There are no universally accepted tools to evaluate operative skills of surgical residents in a timely fashion. The purpose of this study was to determine the feasibility of using a smartphone application, SIMPL (System for Improving and Measuring Procedural Learning), developed by a multi-institutional research collaborative, to achieve a high rate of timely operative evaluations and resident communication and to collect performance data. The authors hypothesized that these goals would be achieved because the process is convenient and efficient. This was a prospective feasibility and engagement study using SIMPL to evaluate residents' operative skills. SIMPL requires the attending surgeon to answer 3 multiple-choice questions: 1) What level of help (Zwisch Scale) was required by the trainee? 2) What was the level of performance? 3) How complex was the case? The evaluator also can dictate a narrative. The sample was composed of 3 faculty members and 3 volunteer senior residents. Predictor variables were the surgeons, trainees, and procedures performed. Outcome variables included number and percentage of procedures performed by faculty-and-resident pairs assessed, time required to complete assessments, time lapsed to submission, percentage of assessments with narratives, and residents' response rates. From March through June 2016, 151 procedures were performed in the operating room by the faculty-and-resident teams. There were 107 assessments submitted (71%). Resident response (self-assessment) to faculty evaluations was 81%. Recorded time to complete assessments (n=75 of 107) was shorter than 2minutes. The time lapsed to submission was shorter than 72hours (100%). Dictations were submitted for 35 evaluations (33%). Data for the type of help, performance, and complexity of cases were collected for each resident. SIMPL facilitates timely intraoperative evaluations of surgical skills, engagement by faculty and residents, and collection of detailed procedural data. Additional prospective trials to assess this tool further are planned.

Uniting Evidence-Based Evaluation with the ACGME Plastic Surgery Milestones: A Simple and Reliable Assessment of Resident Operative Performance

Milestones evaluations in plastic surgery reflect a shift toward competency-based training but have created a number of challenges. The authors have incorporated the smartphone application evaluation tool, System for Improving and Measuring Procedural Learning (SIMPL), that was recently developed by a multi-institutional research collaborative. In this pilot study, the authors hypothesize that SIMPL can improve resident evaluation and also collect granular performance data to simplify compliance with the plastic surgery Milestones. SIMPL was prospectively piloted with a plastic surgery resident and faculty surgeon at Massachusetts General Hospital in this institutional review board-approved study. The study period was a 2-month interval corresponding to the resident's rotation. The resident-faculty combination performed 20 cases together. All cases were evaluated with SIMPL. SIMPL evaluations uniformly took under 1 minute to submit. The average time to completed evaluation from surgery completion was 5 hours (<0.5 hour to 12 hours). Concrete, objective, and specific data about resident performance were collected for every case and presented in a concise format. SIMPL is an innovative, evidence-based evaluation system that makes performance assessment feasible for every procedure in which a plastic surgery resident participates. SIMPL's competency-based framework can be easily scaled to facilitate data collection and reporting of mandatory Milestones evaluations at the program and national levels. This technology will support a shared vocabulary between residents and faculty to enhance intraoperative education.