Assessment Of Procedural Skills Research Articles

Improving assessment of procedural skills in health sciences education: a validation study of a rubrics system in neurophysiotherapy

BackgroundThe development of procedural skills is essential in health sciences education. Rubrics can be useful for learning and assessing these skills. To this end, a set of rubrics were developed in case of neurophysiotherapy maneuvers for undergraduates. Although students found the rubrics to be valid and useful in previous courses, the analysis of the practical exam results showed the need to change them in order to improve their validity and reliability, especially when used for summative purposes. After reviewing the rubrics, this paper analyzes their validity and reliability for promoting the learning of neurophysiotherapy maneuvers and assessing the acquisition of the procedural skills they involve.MethodsIn this cross-sectional and psychometric study, six experts and 142 undergraduate students of a neurophysiotherapy subject from a Spanish university participated. The rubrics’ validity (content and structural) and reliability (inter-rater and internal consistency) were analyzed. The students’ scores in the subject practical exam derived from the application of the rubrics, as well as the rubrics’ criteria difficulty and discrimination indices were also determined.ResultsThe rubrics´ content validity was found to be adequate (Content Validity Index > 0.90). These showed a unidimensional structure, and an acceptable internal consistency (α = 0.71) and inter-rater reliability (Fleiss’ ƙ=0.44, ICC = 0.94). The scores of the subject practical exam practically covered the entire range of possible theoretical scores, showing all the criterion medium-low to medium difficulty indices - except for the one related to the physical therapist position-. All the criterion exhibited adequate discrimination indices (rpbis > 0.39), as did the rubric as a whole (Ferguson’s δ = 0.86). Students highlighted the rubrics´ usefulness for learning the maneuvers, as well as their validity and reliability for formative and summative assessment.ConclusionsThe changed rubrics constitute a valid and reliable instrument for evaluating the execution quality of neurophysiotherapy maneuvers from a summative evaluation viewpoint. This study facilitates the development of rubrics aimed at promoting different practical skills in health-science education.

Development of a Cost-Effective 3D-Printed Airway Suction Simulator for Respiratory Therapy Students.

Three-dimensional (3D)-printed models are cost-effective and can be customized by trainers. This study designed a 3D-printed airway suction simulator for use by respiratory therapy (RT) students. The objective was to demonstrate the cost-effectiveness and application of 3D-printed models in respiratory care training, aiming to enhance the educational experience for RT students. This study developed a 3D-printed airway suction simulator that was cost-effective. A randomized controlled trial was conducted involving RT students to compare effectiveness in a 3D-model group and a control group. Skill assessments and written examinations were used to evaluate the participants' knowledge and skills. A total of 38 second-year RT students were randomly assigned to either the 3D-model group (n = 19) or the control group (n = 19). One participant in the 3D-model group was lost to follow-up during the planned direct observation of procedural skills (DOPS) assessment and satisfaction questionnaire completion. The posttest written examination scores were significantly higher in the 3D-model group than in the control group (100% vs 80%, P = .02). The scores from the DOPS and satisfaction questionnaire were comparable in the 2 groups. This study demonstrated that 3D printing can be used to create a safe and cost-effective airway suction simulator for use by RT students, with potential to enhance training methods. Further research is necessary.

Scaffolding Self-Regulated Learning in Operative Dentistry Through Self-Assessment Training.

This study aimed to investigate how effective self-assessment training is in improving the self-regulated learning (SRL) ability of operative dentistry students. The secondary aim was to explore students' attitudes towards self-assessment in clinical settings. Self-assessment has been the focus of dental education research for decades and has been claimed to be cornerstone in the SRL process. This is quasi-experimental study conducted at Damascus University Faculty of Dental Medicine, Syria. A modified direct observation of procedural skills (DOPS) assessment protocol with a self-assessment component was applied to a convenience sample of students across five clinical encounters in 2022. Data were collected during clinical training on the DOPS form and at the end of the experiment using a structured questionnaire with a single open-ended question. The qualitative part of the study followed a deductive theory-informed content analysis approach, relying on Zimmerman SRL theory. A total number of 32 students participated in the study, eight of whom were male. During the experiment assessment, participants showed overwhelming positive attitudes towards the utility of self-assessment (m ≥ 3.5; max:4) and the overall assessment method (3.3 ≥ m ≥ 3.5; max:4). Cronbach's alpha for the post-experiment questionnaire was 0.888. A total of 84.4% of the participants strongly agreed that self-assessment is useful in clinical training. Data collected (qualitative and quantitative) from participants revealed the positive effects of self-assessment on all three phases of SRL, especially the reflection phase. Self-assessment practice and training showed positive results in terms of improving SRL and performance in a clinical context; SRL appeared to be affected by self-assessment in all phases (forethought, performance, and reflection). Participants' perceptions of self-assessment were generally very positive. These findings imply the potential benefits of applying self-assessment training interventions to foster the SRL ability of learners within the clinical context.

Development and Pilot Testing of a Programmatic System for Competency Assessment in US Anesthesiology Residency Training.

In 2018, a set of entrustable professional activities (EPAs) and procedural skills assessments were developed for anesthesiology training, but they did not assess all the Accreditation Council for Graduate Medical Education (ACGME) milestones. The aims of this study were to (1) remap the 2018 EPA and procedural skills assessments to the revised ACGME Anesthesiology Milestones 2.0, (2) develop new assessments that combined with the original assessments to create a system of assessment that addresses all level 1 to 4 milestones, and (3) provide evidence for the validity of the assessments. Using a modified Delphi process, a panel of anesthesiology education experts remapped the original assessments developed in 2018 to the Anesthesiology Milestones 2.0 and developed new assessments to create a system that assessed all level 1 through 4 milestones. Following a 24-month pilot at 7 institutions, the number of EPA and procedural skill assessments and mean scores were computed at the end of the academic year. Milestone achievement and subcompetency data for assessments from a single institution were compared to scores assigned by the institution's clinical competency committee (CCC). New assessment development, 2 months of testing and feedback, and revisions resulted in 5 new EPAs, 11 nontechnical skills assessments (NTSAs), and 6 objective structured clinical examinations (OSCEs). Combined with the original 20 EPAs and procedural skills assessments, the new system of assessment addresses 99% of level 1 to 4 Anesthesiology Milestones 2.0. During the 24-month pilot, aggregate mean EPA and procedural skill scores significantly increased with year in training. System subcompetency scores correlated significantly with 15 of 23 (65.2%) corresponding CCC scores at a single institution, but 8 correlations (36.4%) were <30.0, illustrating poor correlation. A panel of experts developed a set of EPAs, procedural skill assessment, NTSAs, and OSCEs to form a programmatic system of assessment for anesthesiology residency training in the United States. The method used to develop and pilot test the assessments, the progression of assessment scores with time in training, and the correlation of assessment scores with CCC scoring of milestone achievement provide evidence for the validity of the assessments.

Assessing Chest Tube Insertion Skills Using a Porcine Rib Model-A Validity Study.

Assessments require sufficient validity evidence before their use. The Assessment for Competence in Chest Tube Insertion (ACTION) tool evaluates proficiency in chest tube insertion (CTI), combining a rating scale and an error checklist. The aim of this study was to collect validity evidence for the ACTION tool on a porcine rib model according to the Messick framework. A rib model, consisting of a porcine hemithorax that was placed in a wooden frame, was used as simulator. Participants were recruited from the departments of surgery, pulmonology, and emergency medicine. After familiarization with the rib model and the equipment, standardized instructions and clinical context were provided. They performed 2 CTIs while being scored with the ACTION tool. All performances were assessed live by 1 rater and by 3 blinded raters using video recordings. Generalizability-analysis was performed and mean scores and errors of both groups on the first performance were compared. A pass/fail score was established using the contrasting groups' method. Nine novice and 8 experienced participants completed the study. Generalizability coefficients where high for the rating scale (0.92) and the error checklist (0.87). In the first CTI, novices scored lower than the experienced group (38.1/68 vs. 47.1/68, P = 0.042), but no difference was observed on the error checklist. A pass/fail score of 44/68 was established. A solid validity argument for the ACTION tool's rating scale on a porcine rib model is presented, allowing formative and summative assessment of procedural skills during training before patient contact.

Innovations in assessment in health professions education during the COVID-19 pandemic: A scoping review.

After an initial period of uncertainty during the COVID-19 pandemic, medical universities responded to the pandemic by innovating their methods of teaching-learning and assessment. This scoping review aimed to identify innovations in assessment in medical, dental and nursing education at the undergraduate and postgraduate levels during the pandemic. This review was conducted utilising Arksey and O'Malley's framework where three electronic databases-PubMed, Embase and Web of Science-were systematically searched in February 2022 for articles in English describing innovations in assessments published in or after January 2020. A total of 70 articles were included in the review. Among the primary research articles, 82.1% reported online assessment, 76.1% were related to medical speciality, 74.6% described assessment at the undergraduate level, 68.7% described summative assessment and 58.2% reported assessment of practical skills. All assessments of theoretical knowledge were done virtually with MCQ being the most used assessment tool and virtual OSCE being the most popular assessment tool to assess practical skills; however, it was acknowledged that the assessment of procedural skills was its limitation. Although the studies described the newer forms of assessments as feasible and acceptable, few studies reported the reliability and validity of the assessments. This scoping review identified several innovations in assessment during the COVID-19 pandemic with an overall shift towards virtual or hybrid forms of assessment. However, there was a dearth of evidence regarding the effectiveness of these assessment tools suggesting a need for more research before the tools can be implemented post-pandemic.

Teaching Bundle Care Approach to Intensive Care Nurses to Prevent Ventilator Associated Pneumonia with a Stratified Validated Module.

Module-based teaching of ventilator-associated pneumonia (VAP) with a systematic, stratified approach is expected to have a promising role in teaching. We hypothesized it to improve the skill and knowledge of intensive care unit nurses. The aim of this study was to determine the efficacy of the teaching module with debriefing sessions on the VAP bundle care approach, and the secondary objective was to estimate the improvement in individual components of the VAP bundle. A total of 200 paramedical staff were exposed to a teaching module consisting of pretest didactic lectures, debriefing sessions, reflection by participants, and feedback. Posttest analysis was done to assess knowledge. Skill assessment was done with directly observed procedural skills (DOPS) assessment, and feedback was taken from participants. Follow-up was done at 6 months to assess decay in knowledge and skills. Preworkshop and postworkshop DOPS scores were analyzed using the Mann-Whitney U test. Subgroup analysis was performed using the paired t test. Median pretest and posttest scores were 6 (interquartile range, 4-8) and 13 (interquartile range, 11-15), respectively (P < .001). Comparing DOPS scores before and after exposure to the teaching module, the number of subjects with scores that were below expectations was 24 and 4; meeting expectations was 94 and 24; borderline expectations were 36 and 104; and above expectations was 46 and 58, respectively, between the groups (P < .001). A decay in knowledge and skills was noted in the follow-up. A validated teaching module with debriefing sessions is useful in training bundle care approaches to intensive care unit nurses.

Competency-based assessment of procedural skills in resident training.

No standardized summative tools exist to assess competency in bedside procedures or provide residents and programs with summative feedback. To provide competency-based procedure training and feedback to residents, we created a procedure competency committee (PCC). Here, we describe the PCC process, its impact on procedure training, and examine residents' attainment of competency in bedside procedures. The PCC consisted of hospitalists and met twice annually to review resident procedure portfolios for three academic years 2019-2022 at a university-based internal medicine residency program. Residents were designated to one of the five competency levels; being able to participate, perform under supervision with assistance, perform under direct supervision (DS) without assistance, perform independently with indirect supervision (IS), and perform independently with IS and supervise others. We analyzed the probability of advancing competency levels with each additional procedure using multinomial logistic regression models. Of the 97 residents, 48 (49.5%) were women and 60 (62%) subsequently matched in procedure-oriented fields. More residents achieved IS level for paracentesis than for lumbar puncture (LP) or central venous catheterization (CVC) (62 vs. 25 and 37, respectively; p < .001). Each incremental procedure performed was associated with a higher chance of being advanced to IS group from the DS group; 1.31 times for paracentesis (95% confidence interval [CI] = 1.07, 1.60; p < .008), 1.83 times for LP (95%CI = 1.35, 2.47; p = .0001), and two times for CVC (95%CI = 1.32, 3.05; p = .001). The PCC provided competency-based assessment of a resident's procedural skills and may be used to assess the impact of curriculum changes.

Algorithms for the adaptive assessment of procedural knowledge and skills.

Procedural knowledge space theory (PKST) was recently proposed by Stefanutti (British Journal of Mathematical and Statistical Psychology, 72(2) 185-218, 2019) for the assessment of human problem-solving skills. In PKST, the problem space formally represents how a family of problems can be solved and the knowledge space represents the skills required for solving those problems. The Markov solution process model (MSPM) by Stefanutti et al. (Journal of Mathematical Psychology, 103, 102552, 2021) provides a probabilistic framework for modeling the solution process of a task, via PKST. In this article, three adaptive procedures for the assessment of problem-solving skills are proposed that are based on the MSPM. Beside execution correctness, they also consider the sequence of moves observed in the solution of a problem with the aim of increasing efficiency and accuracy of assessments. The three procedures differ from one another in the assumption underlying the solution process, named pre-planning, interim-planning, and mixed-planning. In two simulation studies, the three adaptive procedures have been compared to one another and to the continuous Markov procedure (CMP) by Doignon and Falmagne (1988a). The last one accounts for dichotomous correct/wrong answers only. Results show that all the MSP-based adaptive procedures outperform the CMP in both accuracy and efficiency. These results have been obtained in the framework of the Tower of London test but the procedures can also be applied to all psychological and neuropsychological tests that have a problem space. Thus, the adaptive procedures presented in this paper pave the way to the adaptive assessment in the area of neuropsychological tests.

Automatic Assessment of Procedural Skills Based on the Surgical Workflow Analysis Derived from Speech and Video.

Automatic surgical workflow analysis (SWA) plays an important role in the modelling of surgical processes. Current automatic approaches for SWA use videos (with accuracies varying from 0.8 and 0.9), but they do not incorporate speech (inherently linked to the ongoing cognitive process). The approach followed in this study uses both video and speech to classify the phases of laparoscopic cholecystectomy, based on neural networks and machine learning. The automatic application implemented in this study uses this information to calculate the total time spent in surgery, the time spent in each phase, the number of occurrences, the minimal, maximal and average time whenever there is more than one occurrence, the timeline of the surgery and the transition probability between phases. This information can be used as an assessment method for surgical procedural skills.

Enhanced simulation-based mastery learning for the attainment of procedural skills competence for acute common care stem (ACCS) trainees

A new curriculum for Acute Common Care Stem (ACCS) trainees was introduced in 2021 and included a mandatory requirement to have specific practical assessments performed, known as Direct Observation of Procedural Skills (DOPS). These aimed to provide evidence of competence and enable progression through training. The curriculum change prompted the creation of a novel one-day skills-based course in order to ensure that all the ACCS trainees met the required level of competence. An enhanced simulation-based mastery learning method was used to ensure that trainees have the clinical knowledge to identify when future practical emergency skills are indicated and have the psychomotor skills necessary to perform these procedures safely and efficiently. This study was aimed at exploring the impact of a simulation-based mastery learning (SBML) [1] in a one-day procedural skills course. New curriculum content was reviewed in order to establish the different practical skills required for the second year of ACCS training. Regional trainees were then sent an online survey to ask which specific procedures they would like included on a skills course. There were five procedures requested: chest drain insertion (Seldinger and open), aspiration of air, DC cardioversion, and external cardiac pacing. Meetings with the local training programme director and simulation lead assisted in the formation of the skills day. Faculty were formed by registrars and consultants in Emergency Medicine. Trainees aimed to reach a competence level that would mean they still need in hospital supervision for any future procedures. Eight ACCS trainees attended a one-day skills-based course in May 2022. The course was preceded by multiple online e-learning resources and videos. The SBML day consisted of a demonstration with deliberate practice followed by DOPS assessments using simulation and trainee-focused feedback. Anonymised pre- and post-course questionnaires were completed by all trainees containing questions surrounding their experience and confidence in performing these procedural skills. All trainees achieved the required level of competence needed at their stage in training. Each trainee stated that their confidence levels improved in performing all five procedures after attending the course ( Mean confidence levels in procedures pre- and post-course using Likert scale (1–5) An enhanced SBML method in a one-day course enabled efficient, standardised procedural skills practice and assessment for a group of ACCS trainees. Training, practice, and assessment in necessary curricular competences were achieved together while improving the level of confidence in trainees performing these procedures. 1. Scahill E, Oliver N, Tallentire V, Edgar S, Tiernan J. An enhanced approach to simulation-based mastery learning: optimising the educational impact of a novel, National Postgraduate Medical Boot Camp. Advances in Simulation. 2021;6(1).

Peer-assessment ability of trainees in clinical restorative dentistry: can it be fostered?

ObjectivesThe study was conducted to investigate whether peer-assessment among dental students at the clinical stage can be fostered and become closer to that of experienced faculty assessors.MethodsA prospective pilot study was conducted in 2021 at the Faculty of Dentistry, Damascus University. Sixteen 5th year clinical students volunteered to participate in the study. A modified version of the validated Peer Direct Observation of Procedural Skills (Peer-DOPS) assessment form was used together with a grading rubric. Participants undertook peer-assessment on their colleagues across three encounters. The difference between peers and faculty assessment was the main variable.ResultsThe mean difference between peers and faculty assessment decreased after each encounter with a significant difference and a medium effect size between the first and third encounters (p = 0.016, d = 0.67). Peer-assessment was significantly higher than faculty, however, the overestimation declined with each encounter reducing the difference between peer- and faculty assessment. Peers’ perception of the educational benefit of conducting assessment was overwhelmingly positive, reporting improvements in their own performance.ConclusionThis pilot study provides preliminary evidence that dental students assessment ability of their peers can be fostered and become closer to that of experienced faculty assessment with practice and assessment-specific instruction.

Impact of a Multimodal Simulation-based Curriculum on Endobronchial Ultrasound Skills

BackgroundCurrently there is no consensus on ideal teaching method to train novice trainees in EBUS. Simulation-based procedure training allows direct observation of trainees in a controlled environment without compromising patient safety.ObjectiveWe wanted to develop a comprehensive assessment of endobronchial ultrasound (EBUS) performance of pulmonary fellows and assess the impact of a multimodal simulation-based curriculum for EBUS-guided transbronchial needle aspiration.MethodsPretest assessment of 11 novice pulmonary fellows was performed using a three-part assessment tool, measuring EBUS-related knowledge, self-confidence, and procedural skills. Knowledge was assessed by 20 multiple-choice questions. Self-confidence was measured using the previously validated EBUS–Subjective Assessment Tool. Procedural skills assessment was performed on Simbionix BRONCH Express simulator and was modeled on a previously validated EBUS–Skills and Task Assessment Tool (EBUS-STAT), to create a modified EBUS-STAT based on internal faculty input via the Delphi method. After baseline testing, fellows participated in a structured multimodal curriculum, which included simulator training, small-group didactics, and interactive problem-based learning sessions, followed by individual debriefing sessions. Posttest assessment using the same three-part assessment tool was performed after 3 months, and the results were compared to study the impact of the new curriculum.ResultsThe mean knowledge score improved significantly from baseline to posttest (52.7% vs. 67.7%; P = 0.002). The mean EBUS–Subjective Assessment Tool confidence scores (maximum score, 50) improved significantly from baseline to posttest (26 ± 7.6 vs. 35.2 ± 6.3 points; P < 0.001). The mean modified EBUS-STAT (maximum score, 105) improved significantly from baseline to posttest (44.8 ± 10.6 [42.7%] vs. 65.3 ± 11.4 [62.2%]; P < 0.001). There was a positive correlation (r = 0.81) between the experience of the test participants and the modified EBUS-STAT scores.ConclusionThis study suggests a multimodal simulation-based curriculum can significantly improve EBUS-guided transbronchial needle aspiration–related knowledge, self-confidence, and procedural skills among novice pulmonary fellows. A validation study is needed to determine if skills attained via a simulator can be replicated in a clinical setting.

Exploration of the implementation of direct observation of procedural skill as an instrument for evaluation of clinical skills in dental professional education

Clinical phase of medical education includes understanding theory, mastery of skills and affective learning. The learning pro-cess were evaluated using a workplace-based assessment (WBA) approach. Direct observation of procedural skill (DOPS) was one of the WBA assessment methods. This research used qualitative methods with a phenomenological approach. The study po-pulation were clinical supervision lecturers and students of the clinical phase. The research method began with the pre-research stage; training for lecturers on how to implement DOPS, followed by the research stage; document study, observation, and focus group discussion (FGD). The results of the study found three major themes, namely concepts, ways of implementing, and obs-tacles to implementing DOPS. According to the teaching staff and students at FKG UK Maranatha, DOPS was an assessment of student procedural skills by direct observation of each stage of the procedural work. The teaching staff's perceptions about DOPS affected the way lecturers provide feedback to participants. As conclusion, full observation is needed to generate useful feedback for students. The effectiveness of DOPS is influenced by the availability of patients, the presence of examiners, and the availability of sufficient time to run DOPS.

How We Do It: Remediation Pathways in a Surgical Simulation Curriculum for Competency Improvement.

Validated assessment of procedural knowledge and skills with formative remediation is a foundational part of achieving surgical competency. High-fidelity simulation programs provide a unique area to assess resident proficiency and independence, as well as to assist in identifying residents in need of further practice. While several studies have validated the use of simulation to attain proficiency of specific technical skills, few have validated remediation pathways for their trainees objectively. In this descriptive analysis, we review 2 remediation pathways within our simulation training curricula and how these are used in assessments of resident proficiency. Two methods of remediation were formulated for use in high-fidelity simulation labs. One remediation pathway was a summative process, where ultimate judgment of resident competency was assessed through intra-operative assessments of a holistic skill set. The second remediation pathway was a formative "coaching" process, where feedback is given at several intervals along the pathway towards a specific technical skills competence. All general surgery residents are enrolled in the longitudinal, simulation curricula. Approximately one-third of surgical residents entered into a remediation pathway for either of the high-fidelity simulation curricula. Both residents and faculty expressed support for the summative and formative remediation pathways as constructed. Residents who entered remediation pathways believed it was a beneficial exercise, and the most common feedback was that remediation principles should be expanded to all residents. Interestingly, faculty demonstrated stronger support for the formative coaching feedback model than the summative assessment model. Through the complementary use of both formative and summative remediation pathways, resident competence can be enriched in a constructive, nonpunitive method for self-directed performance improvement. Both trainees and faculty express high satisfaction with programs explicitly organized to ensure that skills are rated through a standardized process.

Becoming virtually real: using the Virtual Environment for Radiotherapy Training (VERT™) platform for the summative assessment of performance in a palliative radiotherapy treatment technique

Abstract Introduction: Direct observation in the radiotherapy clinic remains the primary method for the assessment of procedural skills. But with increasing automation and reductions in clinical placement time during the recent pandemic, the role of summative, simulation-based assessment is being revisited. The aim of this article is to share early experiences of using Virtual Environment for Radiotherapy Training (VERT) in the assessment of a palliative, parallel pair, external beam treatment delivery technique in a new pre-registration undergraduate programme. Methods: Eight first-year students completed a campus-based individual virtual assessment using virtual patient plans representing metastatic disease in the brain and pelvis and late-stage primary lung tumours. Performance was logged on a 25-item checklist for pre and post-procedure checks and treatment delivery tasks. Results: All eight students participated in the assessment with seven students (87·5%) achieving a pass grade at first attempt. The course team observed that participants demonstrated a range of skills and abilities and were able to compare and contrast individual approaches to patient positioning. Specific feed-forward action points were also highlighted as areas for students’ to focus on during their second placement. The project also identified logistical benefits for assessment teams. Conclusions: While these experiences are a single snapshot, a VERT simulation-based summative assessment is feasible and identified benefits included controlled observation and structured feedback on individual performance and scheduling pressure reductions for clinical teams. However, more work is needed to determine the psychometric qualities and predictability of performance in more complex techniques.

Monthly Trauma Training and Simulation Are Associated With Improved Resident Skill and Leadership.

Training for trauma procedures has been limited to infrequent courses with little data on longitudinal performance, and few address procedural and leadership skills with granular assessment. We implemented a novel training program that emphasized an assessment of trauma resuscitation and procedural skills. This study aimed to determine whether this program could demonstrate improvement in both skill sets in surgical trainees over time. This was a prospective, observational study at a Level I trauma center between November 2018 and May 2019. A procedural skill and simulation program was implemented to train and evaluate postgraduate year (PGY) 1-5 residents. All residents participated in an initial course on procedures such as tube thoracostomy and vascular access, followed by a final evaluation. Skills were assessed by the Likert scale (1-5, 5 noting mastery). PGY 3s and above were additionally evaluated on resuscitation. A paired t test was performed on repeat learners. A total of 40 residents participated in the structured procedural skills and simulation program. Following completion of the program, PGY-2 scores increased from a Mdn [interquartile range, IQR] 3.0 [2.5-4.0] to 4.5 [4.2-4.5]. The PGY-3 scores increased from a Mdn [IQR] 3.95 [3.7-4.6] to 4.8 [4.6-5.0]. Eighteen residents underwent repeat simulation training, with Mdn [IQR] score increases in PGY 2s (3.7 [2.5-4.0] to end score 4.47 [4.0-4.5], p = .03) and PGY 3s (3.95 [3.7-4.6] to end score 4.81 [4.68-5.0], p = .04). Specific procedural and leadership skills also increased over time.

UK Joint Advisory Group consensus statements for training and certification in endoscopic retrograde cholangiopancreatography.

Background and study aims Despite the high-risk nature of endoscopic retrograde cholangiopancreatography (ERCP), a robust and standardized credentialing process to ensure competency before independent practice is lacking worldwide. On behalf of the Joint Advisory Group (JAG), we aimed to develop evidence-based recommendations to form the framework of ERCP training and certification in the UK. Methods Under the oversight of the JAG, a modified Delphi process was conducted with stakeholder representation from the British Society of Gastroenterology, Association of Upper Gastrointestinal Surgeons, trainees and trainers. Recommendations on ERCP training and certification were formulated after formal literature review and appraised using the GRADE tool. These were subjected to electronic voting to achieve consensus. Accepted statements were peer-reviewed by JAG and relevant Specialist Advisory Committees before incorporation into the ERCP certification pathway. Results In total, 27 recommendation statements were generated for the following domains: definition of competence (9 statements), acquisition of competence (8 statements), assessment of competence (6 statements) and post-certification support (4 statements). The consensus process led to the following criteria for ERCP certification: 1) performing ≥ 300 hands-on procedures; 2) attending a JAG-accredited ERCP skills course; 3) in modified Schutz 1–2 procedures: achieving native papilla cannulation rate ≥80%, complete bile duct clearance ≥ 70 %, successful stenting of distal biliary strictures ≥ 75 %, physically unassisted in ≥ 80 % of cases; 4) 30-day post-ERCP pancreatitis rates ≤5 %; and 5) satisfactory performance in formative and summative direct observation of procedural skills (DOPS) assessments. Conclusions JAG certification in ERCP has been developed following evidence-based consensus to quality assure training and to ultimately improve future standards of ERCP practice.

132 Faculty Procedural Skills in Academic Emergency Medicine: A Multisite Survey

Emergency physicians must be competent to perform numerous procedures. It is unclear if clinical practice alone is sufficient to maintain procedural skills, particularly for academic faculty who supervise trainees. Formal assessment of procedural skills beyond training is uncommon. There is a paucity of literature on faculty skills attrition or their preferences for skills maintenance. The faculty procedural skills (FPS) survey sought to determine faculty attitudes, beliefs and experience with procedures to inform programming aimed at skills maintenance. This was a multisite, cross-sectional study of emergency medicine (EM) and pediatric EM (PEM) physician faculty at five large US academic EM departments. Subjects were board certified or board eligible in EM or PEM and provided both clinical care and supervision of trainees. Subjects completed the FPS survey online which consisted of three sections: (1) respondent characteristics and experience performing and supervising EM procedures; (2) degree of confidence performing critical procedures; and (3) attitudes about skills attrition and learning preferences. Internal consistency of each survey section was calculated using Cronbach’s alpha. Items related to beliefs and attitudes were graded on a 5-point scale (strongly disagree to strongly agree), with higher values representing greater agreement. Response agreement (with 95% confidence interval [CI]) was calculated by combining “strongly disagree” with “disagree” and “strongly agree” with “agree” responses. A total of 186 faculty completed the survey (44% response rate). Scale reliability was good (Cronbach’s alpha performance = 0.82; supervision = 0.86; confidence = 0.85). All respondents agreed that ongoing procedural skills training is important for the clinical practice of EM. The majority of respondents agreed: that faculty should have opportunities to retrain for uncommon procedures (98%, 95% CI: ±2%); that skills attrition is a concerning problem (91%, 95% CI: ± 4.1%); that skills attrition is a patient safety issue (89%, 95% CI: ± 4.5%); that attrition was more likely in academic settings (93%, 95% CI: ± 3.7%); that faculty should be required to demonstrate procedural competence (83%, 95% CI: ± 5.4%). The majority of respondents (73%, 95% CI: ± 6.4%) personally experienced procedural skill attrition and a similar number (68%, 95% CI: ± 6.7%) observed it in peers. The most preferred method for skills maintenance was simulation (83%, 95% CI: ± 5.4%). The most preferred learning environment was in groups with peers (89%, 95% CI: ± 4.5%). A majority of faculty reported skills attrition, both personally and observed in peers. Respondents also overwhelmingly agreed they should be competent to perform procedures for which they are credentialed. These findings raise questions about the need for technical skill competency assessments for continued clinical practice. Faculty preferred simulation-based learning and peer environments for skills maintenance. Further study is needed to determine the attitudes, beliefs, and experiences of non-academic emergency physicians with procedural skills attrition.

Introduction of Direct Observation of Procedural Skills as Workplace-Based Assessment Tool in Department of Anesthesiology

Abstract Background: The direct observation of procedural skills (DOPS) was introduced for the workplace-based assessment of procedural skills. It offers an opportunity to provide feedback to trainees. This makes DOPS an authentic measure of clinical competence in anesthesiology training. The goal of this study was to assess the perceptions of both trainees and consultants regarding the use of DOPS and to evaluate the performance of anesthesia postgraduate (PG) trainees over consecutive assessments. Materials and Methods: After approval from the ethical committee and sensitization workshop, two exposures of DOPS per trainee were given for three common anesthesia skills as per their years in training. Thereafter anonymous feedback was collected from faculty and trainees to gather their perception regarding DOPS. Consecutive DOPS scores for trainees were analyzed. Data were presented in terms of percentages, mean, and standard deviation. A P value of &lt;0.05 was considered significant. Results: More than 50% of participants were satisfied with the way DOPS was conducted and thought it was feasible for formative assessment. About 80% of participants were of the view that DOPS is helpful for anesthesia training and improving anesthesia procedural skills. Yet only 40%–50% favored the addition of DOPS to the departmental assessment plan. Significant improvement was observed in DOPS scores of PG trainees. Mean DOPS scores of postgraduate trainee year 1, 2, and 3 (JR 1, JR 2, and JR 3) increased from 2.6 to 4.8, 4 to 5.7, and 5.6 to 7, respectively (P &lt; 0.05). Conclusions: DOPS may be considered as a useful tool for workplace-based assessment for anesthesia PG training.