Objective Performance Feedback Research Articles

Prediction of Robotic Anastomosis Competency Evaluation (RACE) metrics during vesico-urethral anastomosis using electroencephalography, eye-tracking, and machine learning

Residents learn the vesico-urethral anastomosis (VUA), a key step in robot-assisted radical prostatectomy (RARP), early in their training. VUA assessment and training significantly impact patient outcomes and have high educational value. This study aimed to develop objective prediction models for the Robotic Anastomosis Competency Evaluation (RACE) metrics using electroencephalogram (EEG) and eye-tracking data. Data were recorded from 23 participants performing robot-assisted VUA (henceforth ‘anastomosis’) on plastic models and animal tissue using the da Vinci surgical robot. EEG and eye-tracking features were extracted, and participants’ anastomosis subtask performance was assessed by three raters using the RACE tool and operative videos. Random forest regression (RFR) and gradient boosting regression (GBR) models were developed to predict RACE scores using extracted features, while linear mixed models (LMM) identified associations between features and RACE scores. Overall performance scores significantly differed among inexperienced, competent, and experienced skill levels (P value < 0.0001). For plastic anastomoses, R2 values for predicting unseen test scores were: needle positioning (0.79), needle entry (0.74), needle driving and tissue trauma (0.80), suture placement (0.75), and tissue approximation (0.70). For tissue anastomoses, the values were 0.62, 0.76, 0.65, 0.68, and 0.62, respectively. The models could enhance RARP anastomosis training by offering objective performance feedback to trainees.

How can complacency mold managerial decisions? The role of perceptions in strategic decision-making

This paper addresses how managers react to attainment discrepancies in their firms’ performance. Scholars have generally argued that only when performance drops below a predetermined aspiration level firms present a search and change behavior in order to fix this dropping performance. In particular, prior research assumes that organizations compare their performance against preset aspiration levels proceeding from their peers’ performance or their own performance in prior years when determining the urgency of engaging in organizational change. However, empirical evidence on this issue is ambiguous and inconclusive. We tackle this puzzle by studying the executives’ complacency (or cognitive interpretations) with objective results of the firm to determine when the company will decide to change and the magnitude of those changes. Using a sample of 137 medium-sized firms, we do find that the combination of objective results with the managerial perception of them, allows us to obtain a better understanding of the performance feedback literature. Thus, organizational change will be (only) enhanced in front of low managerial levels of complacency with organizational results, disregarding the sign of the objective performance feedback obtained by the firm. Moreover, in our research, we go one step further in analyzing several executives’ characteristics that may affect this managerial complacency.

Changing perspectives: enhancing learning efficacy with the after-action review in virtual reality training for police

The After-Action Review (AAR) in Virtual Reality (VR) training for police provides new opportunities to enhance learning. We investigated whether perspectives (bird’s eye & police officer, bird’s eye & suspect, bird’s eye) and line of fire displayed in the AAR impacted the officers’ learning efficacy. A 3 x 2 ANOVA revealed a significant main effect of AAR perspectives. Post hoc pairwise comparisons showed that using a bird’s eye view in combination with the suspect perspective elicits significantly greater learning efficacy compared to using a bird’s eye view alone. Using the line of fire feature did not influence learning efficacy. Our findings show that the use of the suspect perspective during the AAR in VR training can support the learning efficacy of police officers. Practitioner summary: VR systems possess After-Action Review tools that provide objective performance feedback. This study found that reviewing a VR police training scenario from the bird’s eye view in combination with the suspect perspective enhanced police officers’ learning efficacy. Designing and applying the After-Action Review effectively can improve learning efficacy in VR.

Haptic Feedback Based on Movement Smoothness Improves Performance in a Perceptual-Motor Task.

In many training scenarios, and in surgery in particular, feedback is provided to the trainee after the task has been performed, and the assessment is often qualitative in nature. In this paper, we demonstrate the effect of real-time objective performance feedback conveyed through a vibrotactile cue. Subjects performed a mirror-tracing task that requires coordination and dexterity similar in nature to that required in endovascular surgery. Movement smoothness, a characteristic associated with skilled and coordinated movement, was measured by spectral arc length, a frequency-domain measure of smoothness. The smoothness-based performance metric was encoded as a vibrotactile cue displayed on the user's arm. Performance on the mirror tracing task with smoothness-based feedback was compared to position-based feedback (where the subject was alerted when they moved outside the path boundary) and to a no vibrotactile feedback control condition. Subjects receiving smoothness-based feedback altered their task completion strategies, resulting in faster task completion times, but their accuracy was slightly worse overall than the other two groups. In procedures such as endovascular surgery, the reduction of procedure time that could be achieved with smoothness-based feedback training may be advantageous, despite the fact that accuracy was inferior to that observed with no feedback or position-based feedback.

How do managerial perceptions of performance feedback affect innovation?

Although managers’ perceptions are core to the Performance Feedback Theory, few empirical studies measure managerial perceptions of their organization’s performance and theorize on the (in)consistency between perceptual and objective performance feedback. Based on longitudinal survey data of Canadian organizations, we examine how this (in)consistency affects the propensity for innovation in organizations. Our analysis broadly validates that inconsistency between the two types of feedback dampens innovation. Second, positive perceptions strengthen the relationship between positive objective performance feedback and innovation, leading to increased innovation, whereas negative perceptions strengthen the relationship between negative performance feedback and innovation, echoing the problemistic search hypothesis. We also find that perceptions moderate the effect of objective performance feedback differently in the social and historical dimensions as well above and below the aspiration thresholds.

Embracing informed learner self-assessment during debriefing: the art of plus-delta

The healthcare simulation field has no shortage of debriefing options. Some demand considerable skill which serves as a barrier to more widespread implementation. The plus-delta approach to debriefing offers the advantages of conceptual simplicity and ease of implementation. Importantly, plus-delta promotes learners’ capacity for a self-assessment, a skill vital for safe clinical practice and yet a notorious deficiency in professional practice. The plus-delta approach confers the benefits of promoting uptake of debriefing in time-limited settings by educators with both fundamental but also advanced skills, and enhancing essential capacity for critical self-assessment informed by objective performance feedback. In this paper, we describe the role of plus-delta in debriefing, provide guidance for incorporating informed learner self-assessment into debriefings, and highlight four opportunities for improving the art of the plus delta: (a) exploring the big picture vs. specific performance issues, (b) choosing between single vs. double-barreled questions, (c) unpacking positive performance, and (d) managing perception mismatches.

Using Automated Continuous Instrument Tracking to Benchmark Simulated Laparoscopic Performance and Personalize Training

Laparoscopic simulation is widely used in surgical training. However, the impact of training on performance is difficult to assess. Observation is time-intensive and subjective. SurgTrac laparoscopic box-trainer instrument tracking software provides continuous, automated, real-time, objective performance feedback. We used this data to assess the relationship between task attempts and performance. We assessed whether improvement in performance with repetition could be modeled in learning curves that might be used for benchmarking. Anonymized SurgTrac data for performances undertaken between 10/2016 and 05/2019 were retrospectively extracted. The thread transfer task, a basic instrument handling task, was assessed. Task duration and instrument-based metrics were analyzed; total distance travelled by instrument tips, average speed, average acceleration, and the ratio of movements between the left and right hands. Curve estimation regression was used to assess the relationship between attempt number and metrics for pooled data across the entire cohort of users and amongst individual users with ≥50 attempts. Threshold for significance p = 0.05. SurgTrac has generated the largest available database of performances in box trainer simulated tasks with 64,000 activities performed by over 1450 users in 77 countries to date. Data was derived from the unselected world-wide cohort of SurgTrac users. No participants were excluded. Five hundred seventy-eight users performed 13,027 attempts in the thread transfer task. Across the entire cohort, SurgTrac performance metrics were significantly associated with attempt number. Task duration and total distance decreased with attempt number. This benefit persisted across 100 attempts. Ambidexterity increased with attempt number. Individual candidate performance improved in line with predicted learning curves for better performing candidates. We analyzed the largest database of simulated laparoscopic task performances. Performance improves with practice. Using learning curves derived from peer-group performances as benchmarks, users may be regularly and objectively assessed to support personalization of training.

Mitigating Hospital-Onset Clostridioides difficile: Evaluation of a Standardized Environmental Hygiene Program in Eight Hospitals

Background: Despite ongoing efforts over the past 3 decades, hospital-onset Clostridioides difficile infection (HO-CDI) continues to challenge interventions aimed at its prevention and control. We describe the impact of a model environmental services (EVS) program on the incidence of HO-CDI across 8 hospitals that are part of a nationwide integrated health system. Methods: Eight acute-care hospitals with 44–532 beds (mean, 263 beds) in 6 states with stable endemic HO-CDI incidence densities independently implemented identical sporicidal environmental hygiene interventions in 2017. The program combined the use of a hydrogen peroxide/peroxyacetic acid surface disinfectant for all patient-zone hygienic cleaning combined with a structured model EVS cleaning program that included optimized cleaning and disinfection technique, staff training, and auditing with objective performance feedback, which aligned with 2008 HICPAC/CDC categories I and II as well as 2010 CDC Guidance Level II monitoring program recommendations. After a 3-month phase-in, we compared NHSN-reported LabID HO-CDI SIRs for 18 months before and 12 months after implementation of the program. Results were not shared between sites and data were not collated by the authors until a year after the postintervention results were initially available. Multiple possible confounding factors were evaluated and determined not to have identifiably affected the outcome. Results: Mean preintervention HO-CDI SIRs over the 18 months measured ranged from 0.5 to 1.4 (mean, 1.0 for the group). Following the wash-in period, SIRs decreased precipitously in all sites to a mean of 0.42 for the group by the end of 12 months of the intervention. (P &lt; .0001) (Fig. 1). Individual site improvement ranged from 20% to 92% (mean, 57%) (Fig. 2.) Conclusions: Overall, HO-CDI SIRs decreased almost 60% in the study hospitals following daily sporicidal disinfection cleaning of all patient-zone surfaces in association with ongoing programmatic optimization of cleaning practice. As predicted by earlier single-site studies reporting a favorable impact of sporicidal disinfectant cleaning in outbreak settings, this multisite quasi-experimental study has illustrated the substantial potential impact of hospital-wide sporicidal disinfection integrated with objectively sustained optimized thoroughness of cleaning to decrease the incidence of HO-CDI.Funding: NoneDisclosures: None

Wrist Motion Variation between Novices and Experienced Surgeons Performing Simulated Airway Surgery

ObjectiveTo determine whether wrist motion measured by a smartphone application can be used as a performance metric for a simulated airway procedure requiring both wrist and finger dexterity. We hypothesized that this accelerometer application could detect differences between novices and experienced surgeons performing simulated cricothyrotomy.SettingAcademic medical center.Study DesignProspective pilot cohort study.MethodsVoluntary surgeons and nonsurgeons were recruited. After viewing a training video, smartphones with accelerometer applications were attached to both wrists while subjects performed a cricothyrotomy on a validated task trainer. Procedure time and motion parameters, including average resultant acceleration (ARA), total resultant acceleration (TRA), and suprathreshold acceleration events (STAEs), were collected for dominant and nondominant hands. Subjects were stratified by prior experience. Blinded experts scored each performance using Objective Structured Assessment of Technical Skills (OSATS), and t tests were used to compare performance.ResultsThirty subjects were enrolled. Median age was 26 years, and 20 subjects were male. In the dominant hand, significant differences were seen between novice and experienced surgeons in TRA (P = .005) and procedure time (P = .006), while no significant differences were seen in STAEs (P = .42) and ARA (P = .33). In the nondominant hand, all variables were significantly different between the 2 groups: STAEs (P = .012), ARA (P = .007), TRA (P = .004), and procedure time (P = .006).ConclusionsWrist motion measured by a low-cost smartphone application can distinguish between novice and experienced surgeons performing simulated airway surgery. This tool provides cost-effective and objective performance feedback.

Real-time quantitative performance feedback during strength exercise improves motivation, competitiveness, mood, and performance

Providing quantitative feedback on performance in real-time appears to improve performance in a strength training context. Less is known about the associated effects on psychological variables. Fifteen rugby athletes performed a strength training exercise both with and without objective performance feedback provided in real-time. Feedback increased performance, consistent with previous research. Feedback also led to higher ratings of task competitiveness, state motivation, mood, and workload. These findings provide insight into possible underlying mechanisms responsible for the feedback’s facilitative effects on performance. The findings also suggests that providing this type of feedback may have benefits within other industries, where attempts are frequently made to increase adherence to exercise and rehabilitation programs. This research carries important implications for the design and use of technology within both the sports science and healthcare industry.

The Iowa Disinfection Cleaning Project: Opportunities, Successes, and Challenges of a Structured Intervention Program in 56 Hospitals.

OBJECTIVE A diverse group of hospitals in Iowa implemented a program to objectively evaluate and improve the thoroughness of disinfection cleaning of near-patient surfaces. Administrative benefits of, challenges of, and impediments to the program were also evaluated. METHODS We conducted a prospective, quasi-experimental pre-/postintervention trial to improve the thoroughness of terminal room disinfection cleaning. Infection preventionists utilized an objective cleaning performance monitoring system (DAZO) to evaluate the thoroughness of disinfection cleaning (TDC) expressed as a proportion of objects confirmed to have been cleaned (numerator) to objects to be cleaned per hospital policy (denominator)×100. Data analysis, educational interventions, and objective performance feedback were modeled on previously published studies using the same monitoring tool. Programmatic analysis utilized unstructured and structured information from participants irrespective of whether they participated in the process improvement aspects to the program. RESULTS Initially, the overall TDC was 61% in 56 hospitals. Hospitals completing 1 or 2 feedback cycles improved their TDC percentages significantly (P90% for at least 38 months. A survey of infection preventionists found that lack of time and staff turnover were the most common reasons for terminating the study early. CONCLUSION The study confirmed that hospitals using this program can improve their TDC percentages significantly. Hospitals must invest resources to improve cleaning and to sustain their gains. Infect Control Hosp Epidemiol 2017;38:960-965.

Objective Performance Feedback: Is Numerical Accuracy Necessary?

Performance feedback is ubiquitous in Organizational Behavior Management (OBM); yet its essential components are still debated. It has been assumed that performance feedback must be accurate, but this assumption has not been well established. Two experiments were carried out to research feedback accuracy. Experiment 1 was a single-subject design where performance feedback accuracy was manipulated. Results from Experiment 1 suggested feedback may not need to be accurate to improve performance prompting a follow-up study. Experiment 2 was a repeated measures between-groups design with three types of objective feedback: accurate, high (triple) and low (1/3) inaccurate, and no feedback control. Both accurate and tripled feedback significantly improved performance over the control and low-inaccurate feedback groups. Performance feedback may have reduced time off-task across all three feedback conditions compared to the control. Data from performance feedback research need multi-faceted analysis to fully understand how and why performance feedback changes behavior.

The LEAP™ Gesture Interface Device and Take-Home Laparoscopic Simulators: A Study of Construct and Concurrent Validity.

To assess the potential of the LEAP™ infrared motion tracking device to map laparoscopic instrument movement in a simulated environment. Simulator training is optimized when augmented by objective performance feedback. We explore the potential LEAP has to provide this in a way compatible with affordable take-home simulators. LEAP and the previously validated InsTrac visual tracking tool mapped expert and novice performances of a standardized simulated laparoscopic task. Ability to distinguish between the 2 groups (construct validity) and correlation between techniques (concurrent validity) were the primary outcome measures. Forty-three expert and 38 novice performances demonstrated significant differences in LEAP-derived metrics for instrument path distance (P < .001), speed (P = .002), acceleration (P < .001), motion smoothness (P < .001), and distance between the instruments (P = .019). Only instrument path distance demonstrated a correlation between LEAP and InsTrac tracking methods (novices: r = .663, P < .001; experts: r = .536, P < .001). Consistency of LEAP tracking was poor (average % time hands not tracked: 31.9%). The LEAP motion device is able to track the movement of hands using instruments in a laparoscopic box simulator. Construct validity is demonstrated by its ability to distinguish novice from expert performances. Only time and instrument path distance demonstrated concurrent validity with an existing tracking method however. A number of limitations to the tracking method used by LEAP have been identified. These need to be addressed before it can be considered an alternative to visual tracking for the delivery of objective performance metrics in take-home laparoscopic simulators.

A multi-parameter system for use in neonatal resuscitation research and training

Training staff in the resuscitation of neonates is an essential skill; resuscitation efficacy is difficult to quantify but critical to infant survival. Objective assessments of the efficacy of training methods, resuscitation techniques and devices have used concurrent measurements of air flow and air pressure in commercially available manikins. This system also simultaneously measures the force transmitted through the manikin head during simulated resuscitation, as applying excessive force may be deleterious to newborn infants. The overall accuracy of the force plate over the range 0–5 kg was 0.5%; the output was linear; the frequency response sufficiently high and there was no evidence of hysteresis. This system enables comparison of staff groups, resuscitation techniques and devices in an accurate and reproducible manner. Its use could improve training by offering a means of objective performance feedback through a range of parameters. Evaluation of clinical practice may also result in direct patient benefit.

Two‐Provider Technique for Bimanual Uterine Compression to Control Postpartum Hemorrhage

Bimanual uterine compression may provide a reasonable treatment option for controlling or arresting postpartum hemorrhage in areas where access to care and advanced medical interventions are limited. Preliminary evidence suggests that correctly performed bimanual uterine compression is unsustainable for more than a few minutes, despite empirical evidence and practice recommendations that support extended compression for optimal effect. The objective of this study was to examine the impact of team-based (pairs) bimanual uterine compression for managing postpartum hemorrhage from uterine atony. Obstetricians, nurse-midwives, midwifery students, and unskilled birth attendants (N = 30) in Ghana performed bimanual uterine compression using a simulator with objective performance feedback (6 lights) corresponding to pressure sensors on the uterus. For each participant, we tracked the degree and duration of maintained uterine compression. Birth attendants were then paired (15 pairs) and were similarly assessed performing the technique as a team, with 1 attendant providing external compression to the uterine fundus using 2 hands and the other providing internal pressure against the body of the uterus with a single hand. Individuals were unable to fully compress the uterus and maintain compression for more than 150 seconds without fatiguing. All paired teams were able to fully compress the uterus and maintain the compression for the maximum allotted time of 5 minutes. Results suggest that bimanual uterine compression is more effective when performed by a team, with a primary attendant maintaining internal lower uterine segment pressure and monitoring the patient's condition and a partner applying external pressure to the uterine fundus. For cases where access to advanced medical care is limited or delayed, team-applied bimanual uterine compression could enhance the ability to control or arrest postpartum hemorrhage and increase the likelihood of maternal survival.

Evaluation of simulation-based training on the ability of birth attendants to correctly perform bimanual compression as obstetric first aid

Background Maternal mortality from postpartum hemorrhage remains high globally, in large part because women give birth in rural communities where unskilled (traditional birth attendants) provide care for delivering mothers. Traditional attendants are neither trained nor equipped to recognize or manage postpartum hemorrhage as a life-threatening emergent condition. Recommended treatment includes using uterotonic agents and physical manipulation to aid uterine contraction. In resource-limited areas where Obstetric first aid may be the only care option, physical methods such as bimanual uterine compression are easily taught, highly practical and if performed correctly, highly effective. A simulator with objective performance feedback was designed to teach skilled and unskilled birth attendants to perform the technique. Objectives To evaluate the impact of simulation-based training on the ability of birth attendants to correctly perform bimanual compression in response to postpartum hemorrhage from uterine atony. Methods Simulation-based training was conducted for skilled ( N = 111) and unskilled birth attendants ( N = 14) at two regional (Kumasi, Tamale) and two district (Savelugu, Sene) medical centers in Ghana. Training was evaluated using Kirkpatrick's 4-level model. Results All participants significantly increased their bimanual uterine compression skills after training ( p = 0.000). There were no significant differences between 2-week delayed post-test performances indicating retention ( p = 0.52). Applied behavioral and clinical outcomes were reported for 9 months from a subset of birth attendants in Sene District: 425 births, 13 postpartum hemorrhages were reported without concomitant maternal mortality. Conclusions The results of this study suggest that simulation-based training for skilled and unskilled birth attendants to perform bi-manual uterine compression as postpartum hemorrhage Obstetric first aid leads to improved applied procedural skills. Results from a smaller subset of the sample suggest that these skills could potentially lead to improved clinical outcomes and additional study is merited.

Improving environmental hygiene in 27 intensive care units to decrease multidrug-resistant bacterial transmission*

To determine the thoroughness of terminal disinfection and cleaning of patient rooms in hospital intensive care units and to assess the value of a structured intervention program to improve the quality of cleaning as a means of reducing environmental transmission of multidrug-resistant organisms within the intensive care unit. Prospective, multicenter, and pre- and postinterventional study. Intensive care unit rooms in 27 acute care hospitals. Hospitals ranged in size from 25 beds to 709 beds (mean, 206 beds). A fluorescent targeting method was used to objectively evaluate the thoroughness of terminal room cleaning before and after structured educational, procedural, and administrative interventions. Systematic covert monitoring was performed by infection control personnel to assure accuracy and lack of bias. In total, 3532 environmental surfaces (14 standardized objects) were assessed after terminal cleaning in 260 intensive care unit rooms. Only 49.5% (1748) of surfaces were cleaned at baseline (95% confidence interval, 42% to 57%). Thoroughness of cleaning at baseline did not correlate with hospital size, patient volume, case mix index, geographic location, or teaching status. After intervention and multiple cycles of objective performance feedback to environmental services staff, thoroughness of cleaning improved to 82% (95% confidence interval, 78% to 86%). Significant improvements in intensive care unit room cleaning can be achieved in most hospitals by using a structured approach that incorporates a simple, highly objective surface targeting method and repeated performance feedback to environmental services personnel. Given the documented environmental transmission of a wide range of multidrug-resistant pathogens, our findings identify a substantial opportunity to enhance patient safety by improving the thoroughness of intensive care unit environmental hygiene.

Mission Completion Time is Sensitive to Teleoperation Performance during Simulated Reconnaissance Missions with a Micro-Unmanned Aerial Vehicle

In prior research, we found that temporal measures were affected by input device (game controller vs. mouse); however, other measures were not (e.g., number of targets detected, or number of collisions). The main purpose of the present research was to investigate the sensitivity of a temporal performance measure for simulated reconnaissance missions. Instead of measuring number of targets photographed in a fixed amount of time, this study measured time to photograph a set number of targets. Twenty participants went through simulation-based micro-unmanned aerial vehicle operator training, including three skill courses and two reconnaissance missions. Half the participants used a mouse as their input device, whereas the rest used a game controller. Mission completion time was found to be faster when participants used the game controller than when they used the mouse, for both skill missions and reconnaissance missions. Effective simulation-based training for micro-unmanned vehicles operators requires both relevant simulated missions and objective performance feedback and mastery criteria. These results suggest that mission completion time could provide a good performance measure for this purpose.

Improving Cleaning of the Environment Surrounding Patients in 36 Acute Care Hospitals

The prevalence of serious infections caused by multidrug-resistant pathogens transmitted in the hospital setting has reached alarming levels, despite intensified interventions. In the context of mandates that hospitals ensure compliance with disinfection procedures of surfaces in the environment surrounding the patient, we implemented a multihospital project to both evaluate and improve current cleaning practices. Prospective quasi-experimental, before-after, study. Thirty-six acute care hospitals in the United States ranging in size from 25 to 721 beds. We used a fluorescent targeting method to objectively evaluate the thoroughness of terminal room disinfection cleaning before and after structured educational and procedural interventions. Of 20,646 standardized environmental surfaces (14 types of objects), only 9,910 (48%) were cleaned at baseline (95% confidence interval, 43.4-51.8). Thoroughness of cleaning at baseline correlated only with hospital expenditures for environmental services personnel (P = .02). After implementation of interventions and provision of objective performance feedback to the environmental services staff, it was determined that 7,287 (77%) of 9,464 standardized environmental surfaces were cleaned (P < .001). Improvement was unrelated to any demographic, fiscal, or staffing parameter but was related to the degree to which cleaning was suboptimal at baseline (P < .001). Significant improvements in disinfection cleaning can be achieved in most hospitals, without a substantial added fiscal commitment, by the use of a structured approach that incorporates a simple, highly objective surface targeting method, repeated performance feedback to environmental services personnel, and administrative interventions. However, administrative leadership and institutional flexibility are necessary to achieve success, and sustainability requires an ongoing programmatic commitment from each institution.