Real-world impact of the COVID-19 pandemic on the assessment of anaesthesiology residents

Abstract

Editor—The coronavirus disease 2019 (COVID-19) pandemic has forced a sudden and drastic change in anaesthesia resident training, with significantly reduced case volumes1COVIDSurg CollaborativeElective surgery cancellations due to the COVID-19 pandemic: global predictive modelling to inform surgical recovery plans.Br J Surg. 2020; ([Epub June 13, 2020 ahead of print])https://doi.org/10.1002/bjs.11746Crossref Scopus (741) Google Scholar and frequent trainee redeployment to other clinical areas. Additionally, the increased demands related to applying COVID-19 perioperative protocols and personal protective equipment recommendations, social isolation, and concerns of getting ill and infecting family members are all factors impacting the health and well-being of both residents and faculty.2Anwar A. Seger C. Tollefson A. Diachun C.A.B. Tanaka P. Umar S. Medical education in the COVID-19 era: impact on anesthesiology trainees.J Clin Anesth. 2020; 66: 109949Crossref PubMed Scopus (14) Google Scholar Many postgraduate anaesthesia training programmes worldwide have shifted to competency-based medical education curricula, the success of which are contingent upon adequate clinical exposure and timely workplace-based assessments (i.e. ‘what doctors do in practice’) with learner feedback and with close monitoring to gauge learner progression.3Lockyer J. Carraccio C. Chan M.K. et al.Core principles of assessment in competency-based medical education.Med Teach. 2017; 39: 609-616Crossref PubMed Scopus (190) Google Scholar A paucity of formative assessment opportunities during this pandemic is a significant concern for competency-based programmes; however, the real-world impact of this pandemic on trainee experience and assessment has not been documented. As the largest Canadian anaesthesia residency training programme, located in one of the Canadian epicentres for COVID-19, we were interested in gauging the pandemic's effect on learner assessment. We conducted a study that aimed to compare the quantity and quality (as determined by information richness) of resident assessments completed during the COVID-19 pandemic (March 15 to May 15, 2020) with the same time frame in 2019. The primary outcome was the number of assessments completed. Secondary outcomes included time to assessment completion and richness of information provided (entrustment ratings and comments). Concurrent with the launch of a competency-based programme in 2017, the Anesthesiology Department at the University of Toronto created and implemented a novel ‘assessment dashboard’ that captures and visualises resident workplace-based assessments to facilitate reflective learning (i.e. assessment for learning). Assessment data related to entrustable professional activities (EPAs), essential tasks of a discipline, were extracted for residents in postgraduate years 1–2, including completion rates, time to completion, assessment type (direct observation vs case review), entrustment rating, and faculty comments. We excluded data from postgraduate years 3 and above, as they lacked comparative EPA data for 2019. Ethical approval was obtained from the University of Toronto. Data were analysed using descriptive statistics, with continuous variables summarised using means and standard deviations, and discrete variables using proportions. Significant differences between years were examined using two-sample t-tests for continuous variables and χ2 analyses for discrete variables (α=0.05). In 2019, 32 residents requested 350 EPA assessments, whereas 36 residents requested 276 assessments in 2020 (Table 1). Whilst the number of assessments requested per resident declined from 2019 to 2020 (P=0.046), the number of assessments completed per resident did not change (P=0.069) significantly. The number of assessments completed in the moment, as opposed to after the learning encounter, decreased by 17% from 2019 to 2020 (P<0.001). However, the mean time to assessment completion for those completed after the learning encounter also decreased by nearly 2 days from 2019 to 2020 (P=0.013). Whilst there was no change in the assessment type (direct observation vs case review), there was a reduction in the proportion of anaesthesia vs acute care EPAs assessed (2019: 78.7% anaesthesia; 2020: 66.9%; P=0.003). There was no difference in the overall number of assessments rated as ‘entrusted’ between years (P=0.56), nor in the overall entrustment categories assigned (P=0.30). The number of responses to open-ended questions seeking resident strengths and weaknesses was unchanged between 2019 and 2020 (P=0.072). Similarly, the word counts for these responses were unchanged (P=0.10).Table 1Overview of assessment data. Data presented as n (%), or mean [SD].Variable20192020P-valueNumber of assessments requested, n (%)350276— Postgraduate year 1244 (69.7)151 (54.7) Postgraduate year 2106 (30.3)125 (45.3)Number of assessments completed, n (%)301 (86.0)242 (87.7)— Postgraduate year 1212 (86.9)141 (93.4) Postgraduate year 289 (84.0)101 (80.8)Number of assessments requested per resident10.94 [8.11]7.67 [4.93]0.046Number of assessments completed per resident9.41 [7.21]6.72 [4.61]0.069Number of faculty completing assessments, n (assessments/faculty)147 (2.05 [1.72] assessments/faculty)131 (1.85 [1.36] assessments/faculty)0.29Assessment type, n (%)0.77 Direct observation196/301 (65.1)172/242 (71.1) Case review74/301 (24.6)70/242 (28.9) Not specified31/301 (10.3)0/242 (0)Timing of the assessment, n (%)<0.001 In the moment214/301 (71.1)131/242 (54.1) After the learning encounter87/301 (28.9)111/242 (45.9)Mean time to completion (for assessments completed after the learning encounter)4.27 [6.81] days2.34 [3.76] days0.013 Open table in a new tab The COVID-19 pandemic has led to considerably decreased anaesthesia clinical volumes (65–80% decrease across educational sites based on data from a survey of anaesthesia department chiefs across academic sites in Toronto, ON, Canada) and, based on our data, has negatively impacted residents' requests for assessments and the variety of workplace-based assessment data obtained, with fewer anaesthesia and more acute care EPAs being completed. A decrease in one to two EPA assessments requested per month, particularly if primarily anaesthesia based, could have an educational impact should the pandemic period be prolonged. However, despite fewer assessment requests and less EPAs completed ‘in the moment’ of the learning activity, the number of assessments completed per residents remained stable and the time to completion improved. This is important, as studies have shown that prompt documentation of clinical performance improves the accuracy and effectiveness of feedback.4Ramani S. Krackov S.K. Twelve tips for giving feedback effectively in the clinical environment.Med Teach. 2012; 34: 787-791Crossref PubMed Scopus (190) Google Scholar Additionally, despite the decreased anaesthesia clinical exposure and correspondingly decreased assessment requests, our data suggest that the qualitative feedback provided did not deteriorate. This was shown by the consistency of written comments pertaining to strengths and weaknesses in the EPAs completed. Formative assessments are more meaningful to residents when they provide rich narrative feedback.5van der Vleuten C.P.M. Schuwirth L.W.T. Driessen E.W. et al.A model for programmatic assessment fit for purpose.Med Teach. 2012; 34: 205-214Crossref PubMed Scopus (425) Google Scholar The results of this study highlight the importance for learners and residency programmes to monitor closely formative assessment data. For residents, a learner performance dashboard that analyses current assessment data in a visual format can be a powerful metacognitive tool, as it makes them aware of their performance and behaviour, supporting critical reflection.6Jivet I. Scheffel M. Schmitz M. et al.From students with love: an empirical study on learner goals, self-regulated learning and sense-making of learning analytics in higher education.Internet High Educ. 2020; 47: 100758Crossref Scopus (35) Google Scholar Assessment data can also serve to shape learning behaviours and encourage residents to direct their own learning and develop goal-directed learning actions, which can be collaboratively supported by an academic coach or advisor, for a successful educational outcome.7Garrison D.R. Self-directed learning: towards a comprehensive model.Adult Educ Q. 1997; 48: 18-33Crossref Scopus (524) Google Scholar In a disruptive environment, longitudinal coaching relationships can stimulate reflective dialogue and help residents generate personalised learning plans.8Hall A.K. Nousiainen M.T. Campisi P. et al.Training disrupted: practical tips for supporting competency-based medical education during the COVID-19 pandemic.Med Teach. 2020; 42: 756-761Crossref PubMed Scopus (88) Google Scholar For programmes, an assessment dashboard can facilitate identification of missing evidence of competence and identify clinical gaps and changing trends in learning curves,9Boscardin C. Fergus K.B. Hellevig B. Hauer K.E. Twelve tips to promote successful development of a learner performance dashboard within a medical education program.Med Teach. 2018; 40: 855-861Crossref PubMed Scopus (17) Google Scholar which can be targeted with individualised rotation or curriculum changes as restrictions ease. In addition to longitudinal coaching, this study draws attention to the need for programmes to adapt effectively and efficiently residents' day-to-day curricula to maximise their learning and their ability to acquire necessary competencies. Whilst some training experiences could be replaced with simulation sessions during the pandemic,9Boscardin C. Fergus K.B. Hellevig B. Hauer K.E. Twelve tips to promote successful development of a learner performance dashboard within a medical education program.Med Teach. 2018; 40: 855-861Crossref PubMed Scopus (17) Google Scholar workplace-based assessment requires authentic clinical scenarios. Approaches to assessments could be more intentional and clinical experiences purposefully organised to facilitate evidence of achievement of competencies. Many competency-based programmes, including ours, use assessment tools that quantify a faculty member's entrustment of the resident in patient-care-related tasks. Because meaningful supervisor–trainee relationships have been shown to increase the likelihood of entrustment,10Pingree E.W. Huth K. Harper B.D. et al.Encouraging entrustment: a qualitative study of resident behaviors that promote entrustment.Acad Med. 2020; ([Epub May 5, 2020 ahead of print])https://doi.org/10.1097/ACM.0000000000003487Crossref PubMed Scopus (4) Google Scholar strategic pairing of residents and faculty could potentially enable determination of entrustment over a shorter time period. The COVID-19 pandemic has impacted anaesthesia residents' clinical experience and their workplace-based assessments intended to support learning. As such, programmes and residents will have to be more deliberate to create a bespoke learner experience.