Work-based Assessment Research Articles

FP5.4 - Improving Assessment of General Surgical Trainees in Multidisciplinary Team Working as a Capability in Practice: Perceptions of General Surgical Trainees and Trainers

Abstract Aim To explore how Higher Surgical Trainees (HST) and Trainers feel multidisciplinary team (MDT) working as a capability in practice (CiP) can be improved. Methods Within a deanery, 8 General Surgical HST and 4 Consultants were recruited to participate in virtual semi-structured interviews. Once transcribed, reflexive thematic analysis was performed. Ethical approval was granted. Results MDT is not worked into most trainees’ rotas. Participants felt that by formalising MDT, as with other CiP, it could be better assessed. Trainers, half of whom are MDT leads, stated that responsibility for MDT training and assessment should be placed on all consultants, rather than solely the leads. Participants explored how pathways could be instituted for HST to maintain ‘ownership’ of some patients from assessment clinic to MDT meetings to fully engage. In order to build to this level, a baseline of preparing patients but not presenting was suggested by one of the consultants. While, with one exception, no participants favoured a further work-based assessment, it was advocated by all that further clarification from ISCP regarding expectations of trainees at MDT would allow for more comprehensive assessment in the future. Most participants proposed that all members of MDT contributed to an assessment rather than relying on the Multi-Consultant Report. To prevent issues with timing or resources, a shadow or simulated MDT was suggested for HST. Conclusion General HST and consultants expressed concerns with the current way in which MDT is assessed as a CiP however they proposed ways in which it could be improved in the future.

Implementing work-based assessments in a level 7 data analytics digital degree apprenticeship: challenges and lessons for HEIs

This paper explores the incorporation of work-based learning in the Digital and Technology Solutions Specialist (DTSS) Master’s degree apprenticeship programme in a London-based Higher Education Institution. Specifically, innovative work-based assessments are introduced aiming at utilising workplace resources while meeting academic requirements. Challenges highlighted include balancing academic requirements and business needs as well as supporting learners, employers and academic staff in embracing and implementing the work-based nature of the assessments without compromising academic standards and quality of learning. Lessons are drawn for higher education institutions that consider introducing similar programmes.

An exploration of Irish nutrition educators’ experiences of competency-based assessment in nutrition science education

BackgroundCompetency-based assessment (CBA) supports the development and attainment of skills required for the workforce. Little is known about educators’ experience in developing or implementing CBA in nutrition science education or their opinions on how well it captures a student’s preparedness for the workforce. The objective of this study was to explore educators’ experience of CBA in nutrition education in Ireland.MethodGrounded in interpretivism, in-depth, semi-structured, audio-recorded interviews were conducted with 13 educators from five of the ten undergraduate honours degree nutrition programmes across Ireland. Interviews explored experiences of CBA and perception of students training to prepare for the workforce. A reflexive thematic analysis approach was implemented whereby the data were transcribed, inductively coded, and themes identified.ResultsA clear divide was evident between participants who were confident in their understanding of CBA and those who were unsure or had no knowledge of the term. Those with a clear understanding were more involved in programme development and evaluation. Three themes were identified: ‘Assessment process’ including intended learning outcomes, assessment design, and grading systems, ‘Student-centred approach to assessment’ focusing on work-based assessment and preparation for the workforce, and ‘Upskilling educators’ to equip educators with the skills and knowledge for professional development and to foster student success.ConclusionThe importance of CBA as a student-centred approach, supporting them to meet standards expected to practice as a nutrition professional, was the key experience of educators. Training in CBA and inclusion of more authentic assessment may better prepare students for the workforce.

619 Training the Trainees in Robotic Surgery – a UK Pilot Scheme

Abstract Aim Training the trainees in robotic surgery is a specialised and crucial process, given the precision and complexity involved in performing operations using robotic systems. Robotic training for trainees is challenging due to limited access to robotic systems and the lack of training opportunities. We describe how we implemented a training programme for trainees at West Hertfordshire Teaching Hospitals (WHTH) NHS Trust within a year of the consultants starting their robotics programme. Method Specialist trainees at WHTH who would have been offered training in laparoscopic surgery in colorectal, upper gastrointestinal and general surgery were selected to train as console surgeons on the CMR Surgical Versius system. All trainees had been on a first assist course and had experience in assisting in robotic surgery. Training included online modules, metrics-based virtual training, and technical skills assessment. Live operating was supervised by consultant trainers who sat beside the trainee. Results Six trainees ranging from ST4 to ST8 level were selected to partake in the training programme over a period of six months. Operations were broken down into modular parts and included segmental colectomies, anterior resections, cholecystectomy, and inguinal hernia repair. The trainees have participated in over thirty operations. Feedback from consultant trainers was given verbally and through work-based assessments. No adverse events were reported in training cases. Conclusions Robotic surgical training for specialist trainees can be implemented safely and in a timely manner. Objective metrics such as docking time, console time and volumetric hand motion may help with trainee assessment in the future.

Ensuring robust OSCE assessments: A reflective account from a Scottish school of nursing

AimThis paper reflects on the experience of one Scottish University in conducting a face-to-face Objective Structured Examination (OSCE) for large cohorts of student nurses. It outlines the challenges experienced and learning gained.Borton’s model of reflection frames this work due to its simplicity, ease of application and cyclical nature. BackgroundThe theoretical framework for the OSCE is critical thinking, enabling students to apply those skills authentically. OSCE’s are designed to transfer classroom knowledge to clinical practice and offer an authentic work-based assessment. DesignValidity and robustness are key considerations in any assessment and in OSCE, the number of stations that students encounter is important and debated. We used a case-study based OSCE approach initially over four stations and following reflection, changed to one long station with four phases. ResultsIn OSCE examinations, interrater reliability is a necessity, and students expect equity of approach. We identified that despite clear marking criteria, marks were polarised, with students achieving high or low marks with little middle ground. Review of examination papers highlighted that although students’ overall performance was good, some had failed in at least one station, suggesting a four-station approach may skew results. On reflection we hypothesised that using a one station case study-based, phased approach enabled the examiner to build up a more holistic picture of student knowledge and skills. It also provided the student opportunity to develop a rapport with the examiner and standardised patient, thereby putting them more at ease. We argue that this approach is holistic, authentic and student centred. ConclusionsOur experience highlights that a single station, four phase OSCE is preferrable, enabling students to integrate all aspects of the assessment and provides a holistic view of clinical skills and knowledge.

Electronic distance based clinical skills training development in family medicine in Namibia.

Workplace-based evaluation is one of the most important, but challenging aspects of medical education. The aim was to improve the assessment of the rural community-based clinical training for undergraduate 3rd and 4th year family medicine students at the University of Namibia (UNAM) and implement a paperless process. An online module was developed on the Moodle platform to include a study guide, an electronic portfolio, and electronic resources (e-books and apps) to replace the current paper version of the logbook. We explored local resources by engaging with students and clinical trainers on how to best conduct the initial implementation. Engagement also entailed motivating students to actively participate in the implementation process. All 3rd and 4th year community-based education end service (COBES) students are now submitting proof of clinical learning electronically with the use of their phones in their online portfolio and using online resources. In addition, students in the practical family medicine module that has been introduced in the 6th year since 2023 are now also using an electronic portfolio and these assessment tools.Contribution:Overall feedback from students and supervisors indicates a positive atmosphere of learning and constructive feedback on performance from all team members, hopefully improving work-based assessments and ultimately patient care. More members of the primary health care team were involved and the carbon footprint has also been decreased.

The Next Era of Assessment: Can Ensuring High-Quality, Equitable Patient Care Be the Defining Characteristic?

Previous eras of assessment in medical education have been defined by how assessment is done, from knowledge exams popularized in the 1960s to the emergence of work-based assessment in the 1990s to current efforts to integrate multiple types and sources of performance data through programmatic assessment. Each of these eras was a response to why assessment was performed (e.g., assessing medical knowledge with exams; assessing communication, professionalism, and systems competencies with work-based assessment). Despite the evolution of assessment eras, current evidence highlights the graduation of trainees with foundational gaps in the ability to provide high-quality care to patients presenting with common problems, and training program leaders report they graduate trainees they would not trust to care for themselves or their loved ones. In this article, the authors argue that the next era of assessment should be defined by why assessment is done: to ensure high-quality, equitable care. Assessment should place focus on demanding graduates possess the knowledge, skills, attitudes, and adaptive expertise to meet the needs of all patients and ensuring that graduates are able to do this in an equitable fashion. The authors explore 2 patient-focused assessment approaches that could help realize the promise of this envisioned era: entrustable professional activities (EPAs) and resident sensitive quality measures (RSQMs)/TRainee Attributable and Automatable Care Evaluations in Real-time (TRACERs). These examples illustrate how the envisioned next era of assessment can leverage existing and new data to provide precision education assessment that focuses on providing formative and summative feedback to trainees in a manner that seeks to ensure their learning outcomes prepare them to ensure high-quality, equitable patient outcomes.

Clinical supervisors’ experience of a first-time application of entrustable professional activities in clinical supervision of medical students: findings from a Swedish pilot study

BackgroundRecently, all medical universities in Sweden jointly developed a framework for Entrustable Professional Activities (EPAs) for work-based training and assessment. This framework is now being introduced nationally in the new 6-year undergraduate medical programme that directly lead to a licence to practise. When EPAs are introduced, it is of central importance to gain clinical supervisors’ acceptance to apply the framework in their supervision of students. The aim of this study was therefore to investigate how clinical supervisors, not familiar with EPAs, experience clinical supervision using the framework for EPAs.MethodsWe used a purposive sampling to recruit clinical supervisors. They were given written information on EPAs with a selection of suitable EPAs and the Swedish observation rating scale for assessment of autonomy, and they were offered to attend a 30-minute introductory web course. The participants were informed that EPAs were to be tested, and the students were asked to participate. After the study period the clinical supervisors participated in semi-structured interviews. Inductive qualitative content analysis was used to analyse the transcribed interviews.ResultsThree general themes emerged in the qualitative analysis: Promoting Feedback, Trusting Assessments and Engaging Stakeholders. The participants described benefits from using EPAs, but pointed out a need for preparation and adaptation to facilitate implementation. The structure was perceived to provide structured support for feedback, student involvement, entrustment decisions, enabling supervisors to allow the students to do more things independently, although some expressed caution to rely on others’ assessments. Another concern was whether assessments of EPAs would be perceived as a form of examination, steeling focus from formative feedback. To understand the concept of EPA, the short web-based course and written information was regarded as sufficient. However, concern was expressed whether EPA could be applied by all clinical supervisors. Involvement and adaption of the workplace was pointed out as important since more frequent observation and feedback, with documentation requirements, increase the time required for supervision.ConclusionsEPAs were accepted as beneficial, promoting structured feedback and assessments of the students’ autonomy. Preparation of supervisors and students as well as involvement and adaptation of the workplace was pointed out as important.

Investigating the relationship between processes and profit: A work-based assessment of process used in Australian financial planning firms

The research explores relationship dynamics between process and profit in Australian professional practise. We analyse data collected from 134 financial planning firms located in Southeast Queensland as a sample size. The research introduces a complete financial planning process framework designed to measure the impact that process may have on the relationship with firm profit. Quantitative profit data was recorded using Dovetail software to capture results and evidence regression between groups. The research found that firms’ processes are positively associated with profit, and both process and profit contribute to the decreasing influence of firm agency theory. The research suggests that process could be leveraged as an asset to develop commercial advantages. The research may help identify new measures of standard practise, develop the perception of Australian financial firms and assist to reduce barriers of accessing financial services.

Steps to improve the teaching of clinical reasoning in dermatology: A scoping review and proposal.

Clinical reasoning (CR) is an area of active interest since faults in the diagnostic process can result in errors and possibly delays in care or even patient harm. The purpose of this scoping review was to collect information from the medical literature on approaches utilized to teach and assess CR in dermatology, identify gaps, and prepare a proposal on how to enhance the speciality's ability to develop trainee CR skills. We conducted a review of the published literature (1990-2020) from four databases. The initial search yielded 780 papers, and 42 relevant CR publications met inclusion criteria. Demographic, thematic content, theoretical frameworks, continuum of authenticity, competency/milestone, and assessment/educational intervention data were recorded by two screeners. Trainees at different educational levels from 17 different countries have been assessed in the dermatology literature. Most publications were of a single intervention, appeared underpowered, and had small sample sizes. Only two publications examined work-based assessments (use real patients/stimuli). Knowledge-based studies were the dominant theoretical framework with no studies exclusively focused on process-based CR interventions. Simulation was well represented with 23 (55%) investigations. Rigorous studies that examine CR teaching and assessment in dermatology are lacking. Evidence-based best practices for use of work-based assessments, especially direct observation, need to be developed/adapted for dermatology and validated. Dermatology training programs would benefit from longitudinal data on trainee CR development, process-based CR educational programs, metacognition CR exploration specific to skin disease diagnosis, and studies that yield practical recommendations on how to structure multi-faceted assessments that assess CR.

Reflecting on Experiences of Senior Medical Students' External Clinical Teaching Visits in General Practice Placements: A Pilot Study.

Australian general practice training uses external clinical teaching (ECT) visits for formative work-based assessments. ECT visits appoint senior general-practitioners (GPs) observe trainee GPs' consultations, provide feedback, and make performance-enhancing recommendations. As ECT visits are one of the best assessment tools in Australian GP training, there is limited evidence of its use in undergraduate teaching. This study aims to introduce ECT visits and evaluate assessment tools during senior medical students' GP placement. This study included external and internal GP supervisors and twenty-five Chinese and Australian students during GP placements. The supervisors provided structured in-person feedback, while the ECT assessment tool used a standardised, validated feedback platform to assess every component of a consultation. Students' feedback was recorded and collected by both internal and external supervisors, and then semantically analysed by external supervisors. Twenty-five ECT visit feedbacks were collected and analysed semantically. All participating students rated ECT visits excellently and confirmed the relevance of assessment tools for discussions with supervisors to achieve the designed learning outcomes. Chinese students rated the assessment tools as innovative from a cultural perspective and recommended the ECT visit teaching model and assessment tools to their home university, whereas Australian students suggested more ECT visits during GP placements. Time management was a limitation for both the students and supervisors. ECT visit is an innovative placement teaching model and work-based assessment tool for senior medical students' GP placements, and is rated as the most preferred formative assessment tool. The limitations of this study include small group of students/supervisors and lack of patient feedback; however, all of these limitations can be overcome by involving multiple GP clinics in ongoing large-scale study. ECT visits can be introduced quantitatively into students' GP placement curricula to improve clinical reasoning, learning, and quality assurance with assessments during clinical placements.

Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation.

While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018-2019. Participating faculty (n = 77) observed standardized resident-patient encounters and subsequently completed rater assessment forms (RAFs). Participating faculty in longitudinal rater training. Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (β 0.45, CI 0.12-0.78, p 0.01). This persisted despite adjusting for the faculty's entrustment ratings. There were no other significant linguistic differences by gender. Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident's scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.

Variability in Occult Injury Screening Among Siblings/Household Contacts of Physical Abuse Victims.

The objective of this study is to examine radiologic occult injury screening performance/yield among contacts presenting for precautionary medical assessments and assess factors associated with deferred screening. Data were collected retrospectively from charts of contacts younger than 8 years presenting for precautionary evaluation to a level 1 pediatric emergency department January 1, 2018 to March 31, 2023. Demographics, radiologic performance/yield, physical examination, social work-based psychosocial assessment, reasons for deferred imaging, and diagnostic codes were abstracted. Descriptive statistics and χ 2 analysis are reported. Three hundred ninety contacts were identified; 364 (93.3%) were biological siblings. Most (276, 70.8%) were 2 to 8 years old. Statistically significant relationships were identified with age, insurance, and hospital social work assessment and screening. Thirty-four infants (54%) underwent neuroimaging; no studies were abnormal. Of 114 contacts, <2 years old, 97 (85%) underwent skeletal survey (SS); 9 (9%) SS were abnormal. Twenty-seven (24%) returned for follow-up SS; 4 (14.8%) were abnormal. For 2 contacts, an abnormal initial SS was refuted by follow-up imaging. Physical examinations were abnormal for 11% of contacts. Reasons for deferred imaging included contact well appearance, caregiver concerns, and clinician disagreement with indications. Encounter International Classification of Diseases codes varied, commonly reflecting nonspecific screening assessments. Despite national clinical practice guidelines, studies of abusive injury prevalence and radiologic yield among at-risk contacts exposed to unsafe environments are few. Screening evaluations inclusive of physical examination and radiologic studies identify abuse concerns among at-risk contacts. Further study of factors impacting radiologic screening decisions is needed. Considerations to advance epidemiologic research include standardized diagnostic coding and prospective assessment of radiologic yield.

Developing Bespoke High Volume Low Complexity (HVLC) Theatre Lists With a Focus on Training to Address the Impact of COVID-19: A Pilot Study.

Introduction The COVID-19 pandemic has had an unprecedented impact on both healthcare delivery and surgical training. There have been significant efforts to manage the growing elective waiting list backlog whilst addressing the training deficit. We outline a successful pilot high volume low complexity (HVLC) program held at the Croydon Elective Centre between 2021-2022 which aimed to amalgamate training and elective recovery. Methods Two pilot HVLC training lists were carried out in June 2021 and March 2022. Three parallel theatre lists on each date were supervised by a single consultant floor trainer. All lists followed a standard pre-defined HVLC protocol. Trainees and trainers were invited to participate and encouraged to utilize these lists to sign off relevant work-based assessments. HVLC cases included hernia repairs and simple lesion excisions. Patient, theatre staff, and trainee experiences were collated via questionnaires. Results A total of one consultant supervisor, six trainers, and eight trainees participated in the pilot with a total of 34 elective procedures performed on 29 patients. The mean patient age was 52.4 years with 8 out of 29 patients being female. Of these patients 41.4% were American Society of Anaesthesiologists (ASA) Classification one, 51.72% were ASA two and 6.9% were ASA three. No patients to date werereadmitted to the hospital post-operatively or presented with post-operative complications. One hundred percent of trainees felt satisfied with the training and would recommend it to a colleague. Conclusion The training deficit that developed during the first COVID-19 pandemic wave has been compounded by the second and third waves, and trainees are concerned that further waves are anticipated. Returning to operating is vital and our approach has been shown to improve training, whilst maintaining patient safety and accelerating elective waiting list recovery.

TRAINING IN HIP AND KNEE ARTHROPLASTY: ARE THEY DOING ENOUGH?

Orthopaedic surgery is a practical surgical specialization field, the exit exam for registrars remains written and oral. Despite logbook evaluation and surgical work-based assessments, the question remains: can registrars perform elective surgery upon qualification? In South Africa, obstacles to elective surgical training include the trauma workload, financial constraints, fellowships and the Covid pandemic. In hip and knee arthroplasty, new approaches like the direct anterior approach (DAA) and robotic-assisted knee surgery also contributed to the dilution of cases available for registrar training. There are concerns that orthopaedic registrars do not perform enough cases to achieve surgical proficiency.Review of the last 4 years of registrar logbooks in hip and knee arthroplasty surgery performed in a single tertiary academic hospital in South Africa. We included all primary total hip replacements (THR), total knee replacements (TKR) and hemiarthroplasties (HA) done for neck of femur (NOF) fractures between 1 April 2019 and 30 March 2023. Differentiation between registrar assisting, registrar performing with consultant supervision and registrar performing independent surgery was done.990 hip arthroplasties (472 Primary THR, 216 NOF THR, 302 NOF HA) and 316 Primary TKR were performed during the study period. In primary elective THR the posterior approach was dominant and used in 76% of cases. In NOF THA the DAA was dominant used in 98% of cases. Primary TKR robotic-assisted technologies was used in 27% (n=94) cases. Registrars as the primary surgeon were the highest in NOF THA at 70% of cases and the lowest performing TKR at 25%. During 3-month rotations, an average registrar performed 12 (2 TKR and 10 THR) and assisted in 35 (10 TKR and 25 THR) cases.Despite the large number of arthroplasties operations being performed over the last 4-year period, the surgical cases done by registrars are below, the proposed minimal cases to provide surgical proficiency during their training period.

A70Novel high-fidelity Surgical Simulation Educational Program (SSEP) – a systems-based approach for enhanced patient safety

Increased pressures within the NHS, shortened learning hours and disruptions caused by the COVID-19 pandemic has significantly impacted surgical training. Current courses focus on single training levels, but errors occur through systemwide failures [1]. We developed a SSEP targeting cross specialty and team-based patient management, focused on clinical knowledge and skills, cognitive simulation, leadership and human factors based upon real life events to optimize training in a system that lacks time and exposure. A six-month pilot program was designed with monthly sessions, attended by surgical nurses, junior doctors and consultants, covering common surgical scenarios mapped to the ISCP curriculum [2], and topics highlighted through local clinical governance. Immersive interactive sessions were delivered by a dedicated consultant led education team to provide a mixed reality environment for each simulated scenario. Faculty encompassed emergency, anaesthetic and surgical consultants to facilitate realistic multidisciplinary team working. The professional identity and grade of participants were maintained during each simulation to ensure sessions were representative of real-life events with a clear mutual goal to improve knowledge and skills to transpose into safer patient care at all levels of seniority. Sessions combined simulated scenarios with structured teaching, clear learning objectives, detailed peer feedback and opportunity for juniors to complete work-based assessments with consultants. Senior trainees benefited from participation in the design and delivery of sessions providing opportunities to strengthen their leadership skills. We recorded participants pre- and post-simulation confidence, and knowledge, were measured using 5-point Likert scale feedback forms, and multiple-choice questionnaire (MCQ) paper respectively. Two-paired T analysis showed statistically significant improvements in participant confidence across all 4 simulations ( Pooled two-paired T-Test analysis of participant self-rated confidence, in managing clinical scenarios and core topics, before and after participation in 4 high-fidelity simulated scenarios, at the 95% confidence interval (*Significance α = ≤0.05). Our SSEP is an effective strategy for the improvement of trainees’ confidence and knowledge in managing challenging surgical scenarios, which actively contributes to the development of skills for clinical practice. Evaluation of long-term knowledge retention is necessary. Our ambition is to develop the program as a quality improvement project (QIP) that implements learning from significant events and demonstrate improved outcomes in these domains. We endeavour to embed this simulation training across the region. Authors confirm that all relevant ethical standards for research conduct and dissemination have been met. The submitting author confirms that relevant ethical approval was granted, if applicable.

Linking assessment to real life practice – comparing work based assessments and objective structured clinical examinations using mystery shopping

Objective Structured Clinical Examinations (OSCEs) and Work Based Assessments (WBAs) are the mainstays of assessing clinical competency in health professions’ education. Underpinned by the extrapolation inference in Kane’s Validity Framework, the purpose of this study is to determine whether OSCEs translate to real life performance by comparing students’ OSCE performance to their performance in real-life (as a WBA) using the same clinical scenario, and to understand factors that affect students’ performance. A sequential explanatory mixed methods approach where a grade comparison between students’ performance in their OSCE and WBA was performed. Students were third year pharmacy undergraduates on placement at a community pharmacy in 2022. The WBA was conducted by a simulated patient, unbeknownst to students and indistinguishable from a genuine patient, visiting the pharmacy asking for health advice. The simulated patient was referred to as a ‘mystery shopper’ and the process to ‘mystery shopping’ in this manuscript. Community pharmacy is an ideal setting for real-time observation and mystery shopping as staff can be accessed without appointment. The students’ provision of care and clinical knowledge was assessed by the mystery shopper using the same clinical checklist the student was assessed from in the OSCE. Students who had the WBA conducted were then invited to participate in semi-structured interviews to discuss their experiences in both settings. Overall, 92 mystery shopper (WBA) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was 41.7% [IQR 28.3] and significantly lower compared to the OSCE score 80.9% [IQR 19.0] in all participants (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage real-life patients. Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as opposed to an unfamiliar workplace. As suggested by the activity theory, the performance of the student can be driven by their motivation which differed in the two contexts.

Entrustable professional activities (EPAs) for undergraduate medical education – development and exploration of social validity

BackgroundThe development of entrustable professional activities (EPAs) as a framework for work-based training and assessment in undergraduate medical education has become popular. EPAs are defined as units of a professional activity requiring adequate knowledge, skills, and attitudes, with a recognized output of professional labor, independently executable within a time frame, observable and measurable in its process and outcome, and reflecting one or more competencies. Before a new framework is implemented in a specific context, it is valuable to explore social validity, that is, the acceptability by relevant stakeholders.AimThe aim of our work was to define Core EPAs for undergraduate medical education and further explore the social validity of the constructs.Method and materialIn a nationwide collaboration, EPAs were developed using a modified Delphi procedure and validated according to EQual by a group consisting of teachers nominated from each of the seven Swedish medical schools, two student representatives, and an educational developer (n = 16). In the next step, social validity was explored in a nationwide survey. The survey introduced the suggested EPAs. For each EPA, the importance of the EPA was rated, as was the rater’s perception of the present graduates’ required level of supervision when performing the activity. Free-text comments were also included and analyzed.ResultsTen Core EPAs were defined and validated. The validation scores for EQual ranged from 4.1 to 4.9. The nationwide survey had 473 responders. All activities were rated as “important” by most responders, ranging from 54 to 96%. When asked how independent current graduates were in performing the ten activities, 6 to 35% reported “independent”. The three themes of the free text comments were: ‘relevant target areas and content’; ‘definition of the activities’; and ‘clinical practice and learning’.ConclusionTen Core EPAs were defined and assessed as relevant for Swedish undergraduate medical education. There was a consistent gap between the perceived importance and the certainty that the students could perform these professional activities independently at the time of graduation. These results indicate that the ten EPAs may have a role in undergraduate education by creating clarity for all stakeholders.

43681 Comparison Between Work-Based Assessment, Non-Work-Based Assessment, and Simulation- Based Dermatology Training Methods: Results of a Scoping Review

43681 Comparison Between Work-Based Assessment, Non-Work-Based Assessment, and Simulation- Based Dermatology Training Methods: Results of a Scoping Review

Assessing the utility of a novel entrustment-supervision assessment tool.

Work-based assessments (WBAs) are increasingly used to inform decisions about trainee progression. Unfortunately, WBAs often fail to discriminate between trainees of differing abilities and have poor reliability. Entrustment-supervision scales may improve WBA performance, but there is a paucity of literature directly comparing them to traditional WBA tools. The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) is a previously published WBA tool employing an entrustment-supervision scale with strong validity evidence. This pre-/post-implementation study compares the performance of the O-EDShOT with that of a traditional WBA tool using norm-based anchors. All assessments completed in 12-month periods before and after implementing the O-EDShOT were collected, and generalisability analysis was conducted with year of training, trainees within year and forms within trainee as nested factors. Secondary analysis included assessor as a factor. A total of 3908 and 3679 assessments were completed by 99 and 116 assessors, for 152 and 138 trainees in the pre- and post-implementation phases respectively. The O-EDShOT generated a wider range of awarded scores than the traditional WBA, and mean scores increased more with increasing level of training (0.32 vs. 0.14 points per year, p = 0.01). A significantly greater proportion of overall score variability was attributable to trainees using the O-EDShOT (59%) compared with the traditional tool (21%, p < 0.001). Assessors contributed less to overall score variability for the O-EDShOT than for the traditional WBA (16% vs. 37%). Moreover, the O-EDShOT required fewer completed assessments than the traditional tool (27 vs. 51) for a reliability of 0.8. The O-EDShOT outperformed a traditional norm-referenced WBA in discriminating between trainees and required fewer assessments to generate a reliable estimate of trainee performance. More broadly, this study adds to the body of literature suggesting that entrustment-supervision scales generate more useful and reliable assessments in a variety of clinical settings.