Ottawa Crisis Resource Management Global Research Articles

Assessing Team Performance: A Mixed-Methods Analysis Using Interprofessional in situ Simulation.

Optimizing the performance of emergency department (ED) teams impacts patient care, but the utility of current, team-based performance assessment tools to comprehensively measure this impact is underexplored. In this study we aimed to 1) evaluate ED team performance using current team-based assessment tools during an interprofessional in situ simulation and 2) identify characteristics of effective ED teams. This mixed-methods study employed case study methodology based on a constructivist paradigm. Sixty-three eligible nurses, technicians, pharmacists, and postgraduate year 2-4 emergency medicine residents at a tertiary academic ED participated in a 10-minute in situ simulation of a critically ill patient. Participants self-rated performance using the Team Performance Observation Tool (TPOT) 2.0 and completed a brief demographic form. Two raters independently reviewed simulation videos and rated performance using the TPOT 2.0, Team Emergency Assessment Measure (TEAM), and Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS). Following simulations, we conducted semi-structured interviews and focus groups with in situ participants. Transcripts were analyzed using thematic analysis. Eighteen team-based simulations took place between January-April 2021. Raters' scores were on the upper end of the tools for the TPOT 2.0 (R1 4.90, SD 0.17; R2 4.53, SD 0.27, IRR [inter-rater reliability] 0.47), TEAM (R1 3.89, SD 0.19; R2 3.58, SD 0.39, IRR 0.73), and Ottawa GRS (R1 6.6, SD 0.56; R2 6.2, SD 0.54, IRR 0.68). We identified six themes from our interview data: team member entrustment; interdependent energy; leadership tone; optimal communication; strategic staffing; and simulation empowering team performance. Current team performance assessment tools insufficiently discriminate among high performing teams in the ED. Emergency department-specific assessments that capture features of entrustability, interdependent energy, and leadership tone may offer a more comprehensive way to assess an individual's contribution to a team's performance.

Cross-cultural adaptation and validation of two crisis resource management scales

BackgroundSimulation training programs in crisis resource management must be evaluated using valid and reliable instruments. We translated into Spanish and linguistically validate The Mayo High Performance Teamwork Scale (MHPTS) and Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS) non-technical skills assessment instruments. MethodWe performed a standardised cross-cultural adaptation process. The psychometric properties of both instruments in their versions adapted to Spanish were subsequently evaluated in a sample of 100 students by using exploratory factor analysis and assessing internal consistency and convergent validity through a total of 94 simulation scenarios in urgent medical situations ResultsOur results for the MHPTS showed a one-dimensional structure containing 8 items which explained a total variance of 72.84%; the Ottawa GRS also had a one-dimensional structure, this time with 5 items, which explained a total variance of 91.79%. According to the Cronbach alpha, the internal consistency for the MHPTS was 0.94 (1–8 items) and 0.98 for the Ottawa GRS. In addition, there was a strong correlation between the MHPTS and Ottawa GRS (r = 0.97; p < 0.001). ConclusionWe found strong evidence for the high validity and reliability of the Spanish versions of both these tools when tested in Spanish simulated emergency contexts.

Are battlefield and prehospital trauma scenarios an effective educational tool to teach leadership and crisis resource management skills to undergraduate medical students?

IntroductionLeadership and crisis resource management (CRM) skills are important skills for doctors, however there is a recognised lack of undergraduate leadership education. There remains debate over how best to teach...

An Italian version of the Ottawa Crisis Resource Management Global Rating Scale: a reliable and valid tool for assessment of simulation performance.

Objective measurement of simulation performance requires a validated and reliable tool. However, no published Italian language assessment tool is available. Translation of a published English language tool, the Ottawa Crisis Resource Management Global Rating Scale (GRS), may lead to a validated and reliable tool. After developing an Italian language translation of the English language tool, the study measured the reliability of the new tool by comparison with the English language tool used independently in the same simulation scenarios. In addition, the validity of the Italian language tool was measured by comparison to a skills score also applied independently. The correlation coefficient between the Italian language overall GRS and the English language overall GRS was 0.82 (adjusted 95% confidence interval: 0.62-0.92). The correlation coefficient between the Italian language overall GRS and the skill score was 0.85 (adjusted 95% confidence interval 0.68-0.94). This study demonstrated that the Italian language GRS has acceptable reliability when compared with the English language tool, suggesting that it can be used reliably to evaluate the performance during simulated emergencies. The study also suggests that the tool has acceptable validity for assessing the simulation performance. The study suggests that the Italian language GRS translation has reasonable reliability when compared with the English language GRS and reasonable validity when compared with the assessment of the skills scores. Data suggest that the instrument is adequately reliable for informal and formative type of examinations, but may require further confirmation before use for high-stake examinations such as licensing.

Simulation Testing for Selection of Critical Care Medicine Trainees. A Pilot Feasibility Study.

Selection of physicians into anesthesiology, intensive care, and emergency medicine training has traditionally relied on evaluation of curriculum vitae, letters of recommendation, and interviews, despite these methods being poor predictors of subsequent workplace performance. In this study, we evaluated the feasibility and face validity of incorporating assessment of nontechnical skills in simulation and personality traits into an existing junior doctor selection framework. Candidates short-listed for a critical care residency position were invited to participate in the study. On the interview day, consenting candidates participated in a simulation scenario and debriefing and completed a personality test (16 Personality Factor Questionnaire) and a survey. Timing of participants' progression through the stations and faculty staff numbers were evaluated. Nontechnical skills were evaluated and candidates ranked using the Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS). Nontechnical skills ranking and traditional selection method ranking were compared using the concordance correlation coefficient. Interrater reliability was assessed using the concordance correlation coefficient. Thirteen of 20 eligible participants consented to study inclusion. All participants completed the necessary stations without significant time delays. Eighteen staff members were required to conduct interviews, simulation, debriefing, and personality testing. Participants rated the simulation station to be acceptable, fair, and relevant and as providing an opportunity to demonstrate abilities. Personality testing was rated less fair, less relevant, and less acceptable, and as giving less opportunity to demonstrate abilities. Participants reported that simulation was equally as stressful as the interview, whereas personality testing was rated less stressful. Assessors rated both personality testing and simulation as acceptable and able to provide additional information about candidates. The Ottawa GRS showed moderate interrater concordance. There was moderate concordance between rankings based on traditional selection methods and Ottawa GRS rankings (ρ = 0.52; 95% confidence interval, -0.02 to 0.82; P = 0.06). A multistation selection process involving interviews, simulation, and personality testing is feasible and has face validity. A potential barrier to adoption is the high number of faculty required to conduct the process.

Exploring intensive care nurses' team performance in a simulation-based emergency situation, - expert raters' assessments versus self-assessments: an explorative study.

BackgroundEffective teamwork has proven to be crucial for providing safe care. The performance of emergencies in general and cardiac arrest situations in particular, has been criticized for primarily focusing on the individual’s technical skills and too little on the teams’ performance of non-technical skills. The aim of the study was to explore intensive care nurses’ team performance in a simulation-based emergency situation by using expert raters’ assessments and nurses’ self-assessments in relation to different intensive care specialties.MethodsThe study used an explorative design based on laboratory high-fidelity simulation. Fifty-three registered nurses, who were allocated into 11 teams representing two intensive care specialties, participated in a videotaped simulation-based cardiac arrest setting. The expert raters used the Ottawa Crisis Resource Management Global Rating Scale and the first part of the Mayo High Performance Teamwork Scale to assess the teams’ performance. The registered nurses used the first part of the Mayo High Performance Teamwork Scale for their self-assessments, and the analyses used were Chi-square tests, Mann–Whitney U tests, Spearman’s rho and Intraclass Correlation Coefficient Type III.ResultsThe expert raters assessed the teams’ performance as either advanced novice or competent, with significant differences being found between the teams from different specialties. Significant differences were found between the expert raters’ assessments and the registered nurses’ self-assessments.ConclusionsTeams of registered nurses representing specialties with coronary patients exhibit a higher competence in non-technical skills compared to team performance regarding a simulated cardiac arrest. The use of expert raters’ assessments and registered nurses’ self-assessments are useful in raising awareness of team performance with regard to patient safety.Electronic supplementary materialThe online version of this article (doi:10.1186/s12912-014-0047-5) contains supplementary material, which is available to authorized users.

Heart rate, anxiety and performance of residents during a simulated critical clinical encounter: a pilot study.

BackgroundHigh-fidelity patient simulation has been praised for its ability to recreate lifelike training conditions. The degree to which high fidelity simulation elicits acute emotional and physiologic stress among participants – and the influence of acute stress on clinical performance in the simulation setting – remain areas of active exploration. We examined the relationship between residents’ self-reported anxiety and a proxy of physiologic stress (heart rate) as well as their clinical performance in a simulation exam using a validated assessment of non-technical skills, the Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS).MethodsThis was a prospective observational cohort study of emergency medicine residents at a single academic center. Participants managed a simulated clinical encounter. Anxiety was assessed using a pre- and post-simulation survey, and continuous cardiac monitoring was performed on each participant during the scenario. Performance in the simulation scenario was graded by faculty raters using a critical actions checklist and the Ottawa GRS instrument.ResultsData collection occurred during the 2011 academic year. Of 40 eligible residents, 34 were included in the analysis. The median baseline heart rate for participants was 70 beats per minute (IQR: 62 – 78). During the simulation, the median maximum heart rate was 140 beats per minute (IQR: 137 – 151). The median minimum heart rate during simulation was 81 beats per minute (IQR: 72 – 92), and mean heart rate was 117 beats per minute (95% CI: 111 – 123). Pre- and post-simulation anxiety scores were equal (mean 3.3, IQR: 3 to 4). The minimum and maximum Overall Ottawa GRS scores were 2.33 and 6.67, respectively. The median Overall score was 5.63 (IQR: 5.0 to 6.0). Of the candidate predictors of Overall performance in a multivariate logistic regression model, only PGY status showed statistical significance (P = 0.02).ConclusionsSimulation is associated with physiologic stress, and heart rate elevation alone correlates poorly with both perceived stress and performance. Non-technical performance in the simulation setting may be more closely tied to one’s level of clinical experience than to perceived or actual stress.

Simone-simulation in medical oncology education: Part I—Pilot feasibility study.

TPS6638 Background: Advances in medical oncology therapeutics have led to increasingly challenging cancer scenarios (CCS), including an expanding spectrum of treatment side effects and chronic cancer complications. There is no defined curriculum addressing CCS management. Simulation based training with debriefing (SBTD) is an educational method utilizing a virtual medium to mimic clinical scenarios. We hypothesize that SBTD, as an educational tool, is better than traditional didactic teaching of CCS management. This unique study tests the feasibility of high-fidelity SBTD in medical oncology education. We eventually aim to develop a national standardized oncology SBTD curriculum to train oncologists in CCS management Methods: With ethics approval, a curriculum highlighting CCS topics was created. Three clinical scenarios were developed and programmed using the high-fidelity SimMan mannequin. Scenarios last 10 minutes, and participants’ decisions determine the course of the scenario. Participants are recruited from medical oncology and internal medicine. After receiving the curriculum, participant demographics are collected and they are randomized 1:1 to intervention Arm A or B. Both arms perform three simulation scenarios. After scenario #1, all participants take a quiz testing CCS-relevant knowledge. Then, Arm A receives an expert-facilitated debriefing; Arm B receives a didactic lecture covering CCS management. The next day all participants perform simulation and quiz #2, with simulation and quiz #3 planned for 8 weeks later. Each simulation is videotaped for two independent reviewers to grade performance using the validated Ottawa Crisis Resource Management Global Rating Scale. Beyond feasibility, outcomes include change in performance and quiz scores, and participants’ satisfaction with educational method as assessed by questionnaire after simulation #2-3. Differences between the three simulation scores in both arms will be calculated, and assessed using independent t-test. At this time, eleven participants have been enrolled and all participants have completed two simulations. Data will be analyzed after the third simulation.

Emergency medicine resident crisis resource management ability: a simulation-based longitudinal study

BackgroundSimulation has been identified as a means of assessing resident physicians’ mastery of technical skills, but there is a lack of evidence for its utility in longitudinal assessments of residents’ non-technical clinical abilities. We evaluated the growth of crisis resource management (CRM) skills in the simulation setting using a validated tool, the Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS). We hypothesized that the Ottawa GRS would reflect progressive growth of CRM ability throughout residency.MethodsForty-five emergency medicine residents were tracked with annual simulation assessments between 2006 and 2011. We used mixed-methods repeated-measures regression analyses to evaluate elements of the Ottawa GRS by level of training to predict performance growth throughout a 3-year residency.ResultsOttawa GRS scores increased over time, and the domains of leadership, problem solving, and resource utilization, in particular, were predictive of overall performance. There was a significant gain in all Ottawa GRS components between postgraduate years 1 and 2, but no significant difference in GRS performance between years 2 and 3.ConclusionsIn summary, CRM skills are progressive abilities, and simulation is a useful modality for tracking their development. Modification of this tool may be needed to assess advanced learners’ gains in performance.

Board 348 - Research Abstract Validation of the Mayo High Performance Team Work Scale and Ottawa Crisis Resource Management Global Rating Scale for Evaluation of Crisis Team Training (Submission #472)

Board 348 - Research Abstract Validation of the Mayo High Performance Team Work Scale and Ottawa Crisis Resource Management Global Rating Scale for Evaluation of Crisis Team Training (Submission #472)

A Comparison of Global Rating Scale and Checklist Scores in the Validation of An Evaluation Tool to Assess Performance in the Resuscitation of Critically Ill Patients during Simulated Emergencies.

BACKGROUND: Resuscitation of critically ill patients requires medical knowledge, clinical skills and non-medical skills, referred to as crisis resource management (CRM) skills. Most human errors that occur in medical crises are attributed not to deficits in medical knowledge, but rather to errors in non-medical skills such as CRM. Few opportunities currently exist to formally evaluate performance during resuscitation. Furthermore, no gold standard exists for evaluation of CRM performance. A pilot study using mannequin-based human patient simulation examined CRM performance during simulated emergencies, and examined the validity of a novel rating instrument in evaluating CRM performance - the Ottawa Crisis Resource Management Global Rating Scale (abbreviated as “Ottawa GRS”). Debate exists as to whether checklists or global rating scales are superior in formal evaluation of performance. In this study, a checklist for CRM performance and the Ottawa GRS were used to evaluate CRM performance. Both instruments were peer-reviewed, and divided into five categories of CRM skills. The Ottawa GRS provided scores for each category along a seven-point Likert scale, and also included a separate score for overall performance. The CRM checklist provided measurements of 15 individual items for the five categories of CRM, with a cumulative score of 30 points (two points per item). METHODS: First and third-year residents participated in two simulator scenarios, each recreating emergencies commonly observed in acute care settings. Using edited video recordings of each session, three blinded raters evaluated resident performance with both the Ottawa GRS and the CRM checklist. Validity of each rating instrument was measured on the basis of content validity, response process, internal structure and response to other variables. Response to variables was measured in this study by the response to the variable of level of training. T-test analysis of Ottawa GRS and CRM checklist scores were conducted to examine response to level of training. Internal structure was measured in part by measures of inter-rater reliability. Intra-class correlation coefficient (ICC) scores were used to measure inter-rater reliability for both scenarios. RESULTS: A total of 32 first-year and 28 third-year residents were recruited into the study during a 24-month period. Both the Ottawa GRS and the CRM checklist demonstrated the ability to discriminate between levels of training in all categories examined (p <0.0019 to p <0.0001). This difference was noted with all raters, and with each scenario. No statistically significant difference in resident performance was noted between the first and second scenario. Intra-class correlation coefficient (ICC) scores for Ottawa GRS overall performance scores were 0.590 and 0.613 for each scenario. ICC scores for CRM checklist cumulative scores were 0.633 and 0.545 for each scenario. All raters in the study indicated a strong preference for the Ottawa GRS due to ease of administration and scoring. Raters also indicated the Ottawa GRS was superior in providing the opportunity to rate overall performance. CONCLUSIONS: Both the Ottawa GRS and CRM checklist demonstrated the ability to differentiate CRM performance based on levels of training. Both instruments demonstrated acceptable measures of inter-rater reliability. The Ottawa GRS was strongly preferred by raters for its ease of use and administration. The Ottawa GRS appears well-suited as a formal evaluation tool to rate physician performance in CRM during simulated emergencies.

A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study

Resuscitation of critically ill patients requires medical knowledge, clinical skills, and nonmedical skills, or crisis resource management (CRM) skills. There is currently no gold standard for evaluation of CRM performance. The primary objective was to examine the use of high-fidelity simulation as a medium to evaluate CRM performance. Since no gold standard for measuring performance exists, the secondary objective was the validation of a measuring instrument for CRM performance-the Ottawa Crisis Resource Management Global Rating Scale (or Ottawa GRS). First- and third-year residents participated in two simulator scenarios, recreating emergencies seen in acute care settings. Three raters then evaluated resident performance using edited video recordings of simulator performance. A Canadian university tertiary hospital. : The Ottawa GRS was used, which provides a 7-point Likert scale for performance in five categories of CRM and an overall performance score. Construct validity was measured on the basis of content validity, response process, internal structure, and response to other variables. One variable measured in this study was the level of training. A t-test analysis of Ottawa GRS scores was conducted to examine response to the variable of level of training. Intraclass correlation coefficient scores were used to measure interrater reliability for both scenarios. Thirty-two first-year and 28 third-year residents participated in the study. Third-year residents produced higher mean scores for overall CRM performance than first-year residents (p < .0001) and in all individual categories within the Ottawa GRS (p = .0019 to p < .0001). This difference was noted for both scenarios and for each individual rater (p = .0061 to p < .0001). No statistically significant difference in resident scores was observed between scenarios. Intraclass correlation coefficient scores of .59 and .61 were obtained for scenarios 1 and 2, respectively. Data obtained using the Ottawa GRS in measuring CRM performance during high-fidelity simulation scenarios support evidence of construct validity. Data also indicate the presence of acceptable interrater reliability when using the Ottawa GRS.