Assessment Of Procedural Skills Research Articles

Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta.

Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE. This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard. Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed. Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts. Diagnostic test, no or poor gold standard, level V.

Comparison of Two Simulated Procedural Assessment Formats in Attending Emergency Physicians.

BackgroundEmergency physicians must be proficient at inserting central venous catheters and performing lumbar punctures to provide life-saving therapies to critically ill patients. An assessment of procedural skill is rarely performed after an emergency physician has completed residency. Current board certification exams for emergency medicine focus only on verbal descriptions of procedures to assess skill. We compared two methods of procedural skill assessment, simulated task trainer and verbal description, to assess the range of skill in central venous catheter insertion and lumbar punctures of emergency attending physicians at a large, urban, academic tertiary care institution.MethodologyThis is a prospective cohort study of simulated internal jugular central venous catheter insertion and lumbar puncture skill by emergency attending physicians on a task trainer versus verbal description. A total of 17 attending emergency medicine physicians consented to participate in the study during a yearly procedural skills session. For each subject, two expert raters used previously published checklists to assess procedural skill and give a global rating score.ResultsMore checklist items were performed correctly on the task trainer than on verbal assessment for central line (task trainer = 78.4% ± 8.32% and verbal = 68.26% ± 8.9%) and lumbar puncture (task trainer = 85.57% ± 7.6% and verbal = 73.53%4 ± 10.34%) procedures, both with significant differences (p < 0.001). Of the participants, 82% strongly preferred the task trainer format to the verbal description assessment format.ConclusionsThe higher scores on the simulated format compared to the current verbal format imply that a shift towards simulated procedural assessment techniques may benefit examinees. More work is needed to determine if objective checklist scores for practicing attending emergency physicians correlate with subjective expert assessments of their procedural skills.

Determinants of Nursing Students’ Confidence in Peripheral Intravenous Catheter Insertion and Management

BACKGROUND: Peripheral intravenous catheter (PIVC) is one of the medical devices commonly inserted in hospitalized patients, both adults and pediatrics. PIVC has crucial functions for delivering drugs, fluids, blood transfusions, and diagnostic tests for patients. Thus, nursing students must be confident in terms of insertion and management of this device. However, studies assessing nursing students’ confidence and its determinants are still limited. AIM: This study aims to examine the internship nursing student’s confidence in PIVC insertion and management in adult patients and its contributing factors. METHODS: A cross-sectional study was conducted to measure internship nursing students’ confidence in PIVC cannulation and management in adult patients as well as its contributing factors. Purposive samples of 100 nursing students in Yogyakarta were recruited, and a 5-point Likert scale questionnaire consisting of 19 questions was used. Kruskal–Wallis test was utilized to investigate the association of the internship nursing student confidence on PIVC insertion and maintenance with the factors. A general linear regression analysis was performed to obtain adjusted estimates of the potential factors with students’ confidence. RESULTS: Results indicated that the internship nursing student’s confidence score ranged from 57 to 95, with a mean value of 75 (±8.1). Among determinants of the student’s confidence investigated in this study, t-test analysis showed that the students’ confidence was associated with their participation in expert lecture, bedside teaching, and direct observation of procedural skill assessment of PIVC insertion and care (p &lt; 0.05). The general linear analysis showed that only bedside teaching and interaction of bedside teaching and direct observation procedural skill assessment were significant predictors of the internship nursing student’s confidence on PIVC insertion and care (β = 10.99, 95% confidence interval [CI] 2.00–20.00 and β = 13.15, 95% CI 1.20–25.15, p &lt; 0.05, respectively). CONCLUSION: This result indicated that nursing students need direct simulation and assessment of PIVC insertion and care to the patients to improve their confidence in PIVC insertion and management in adult patients.

Development and Pilot Testing of Entrustable Professional Activities for US Anesthesiology Residency Training.

Modern medical education requires frequent competency assessment. The Accreditation Council for Graduate Medical Education (ACGME) provides a descriptive framework of competencies and milestones but does not provide standardized instruments to assess and track trainee competency over time. Entrustable professional activities (EPAs) represent a workplace-based method to assess the achievement of competency milestones at the point-of-care that can be applied to anesthesiology training in the United States. Experts in education and competency assessment were recruited to participate in a 6-step process using a modified Delphi method with iterative rounds to reach consensus on an entrustment scale, a list of EPAs and procedural skills, detailed definitions for each EPA, a mapping of the EPAs to the ACGME milestones, and a target level of entrustment for graduating US anesthesiology residents for each EPA and procedural skill. The defined EPAs and procedural skills were implemented using a website and mobile app. The assessment system was piloted at 7 anesthesiology residency programs. After 2 months, faculty were surveyed on their attitudes on usability and utility of the assessment system. The number of evaluations submitted per month was collected for 1 year. Participants in EPA development included 18 education experts from 11 different programs. The Delphi rounds produced a final list of 20 EPAs, each differentiated as simple or complex, a defined entrustment scale, mapping of the EPAs to milestones, and graduation entrustment targets. A list of 159 procedural skills was similarly developed. Results of the faculty survey demonstrated favorable ratings on all questions regarding app usability as well as the utility of the app and EPA assessments. Over the 2-month pilot period, 1636 EPA and 1427 procedure assessments were submitted. All programs continued to use the app for the remainder of the academic year resulting in 12,641 submitted assessments. A list of 20 anesthesiology EPAs and 159 procedural skills assessments were developed using a rigorous methodology to reach consensus among education experts. The assessments were pilot tested at 7 US anesthesiology residency programs demonstrating the feasibility of implementation using a mobile app and the ability to collect assessment data. Adoption at the pilot sites was variable; however, the use of the system was not mandatory for faculty or trainees at any site.

Introduction of direct observation of procedural skills (DOPS) as a formative assessment tool during postgraduate training in anaesthesiology: Exploration of perceptions.

Background and Aims:Formative assessment of procedural skills of Anaesthesiology postgraduate (PG) students is not conducted conventionally. Direct observation of procedural skills (DOPS) helps to identify gaps in performance and provides structured feedback. The present study was taken to explore perceptions of PG students and faculty about DOPS.Methods:This mixed design interventional study was conducted on 12 PGs and 10 faculty members in Department of Anaesthesiology. After conducting DOPS, a pre-validated questionnaire was given to them about perceptions using 3-point Likert's scale along with open ended questions. Statistical analysis was done using descriptive statistics of perception to calculate percentages and themes were identified for qualitative data.Results:Responses of students were positive about skill improvement (83%), time provided (75%), feedback (100%), interaction (83%), motivation (83%), satisfaction (83%), effectiveness (83%) and opportunity creation (92%). Faculty responded positively regarding change in attitude (100%), effectiveness (100%), scope (90%), feasible application (90%), ease (90%), opportunity creation (80%), gap identification (100%), satisfaction (80%). However, 60% felt training was required, 50% thought more time and commitment was required. Themes identified were DOPS is comprehensive, interactive, student-friendly, good teaching-learning tool, identifies gaps, provides focus for learning, provides systematic constructive feedback, improves skills, prepares for future practice, requires planning, may not reflect competence, has assessor variability and can be included in PG curriculum.Conclusion:DOPS was perceived as an effective assessment and teaching-learning tool by PG students as well as faculty.

Speech-Based Surgical Phase Recognition for Non-Intrusive Surgical Skills' Assessment in Educational Contexts.

Surgeons’ procedural skills and intraoperative decision making are key elements of clinical practice. However, the objective assessment of these skills remains a challenge to this day. Surgical workflow analysis (SWA) is emerging as a powerful tool to solve this issue in surgical educational environments in real time. Typically, SWA makes use of video signals to automatically identify the surgical phase. We hypothesize that the analysis of surgeons’ speech using natural language processing (NLP) can provide deeper insight into the surgical decision-making processes. As a preliminary step, this study proposes to use audio signals registered in the educational operating room (OR) to classify the phases of a laparoscopic cholecystectomy (LC). To do this, we firstly created a database with the transcriptions of audio recorded in surgical educational environments and their corresponding phase. Secondly, we compared the performance of four feature extraction techniques and four machine learning models to find the most appropriate model for phase recognition. The best resulting model was a support vector machine (SVM) coupled to a hidden-Markov model (HMM), trained with features obtained with Word2Vec (82.95% average accuracy). The analysis of this model’s confusion matrix shows that some phrases are misplaced due to the similarity in the words used. The study of the model’s temporal component suggests that further attention should be paid to accurately detect surgeons’ normal conversation. This study proves that speech-based classification of LC phases can be effectively achieved. This lays the foundation for the use of audio signals for SWA, to create a framework of LC to be used in surgical training, especially for the training and assessment of procedural and decision-making skills (e.g., to assess residents’ procedural knowledge and their ability to react to adverse situations).

The Effect of Repeated Direct Observation of Procedural Skills (R-DOPS) Assessment Method on the Clinical Skills of Anesthesiology Residents.

BackgroundThe ultimate result of patient care is one of the most important outcomes in medical education. Several methods, including the direct observation of procedural skills (DOPS), have been proposed to assess professional competencies in clinical practice.ObjectivesThis study aimed to assess the effects of the Repeated DOPS (R-DOPS) method on the performance of procedural skills in anesthesiology residents.MethodsThe procedural skill performance of anesthesiology residents was assessed using a standard DOPS protocol from May to October 2019. Their scores were then objectively recorded, and the satisfaction rates regarding the 2 DOPS exams were assessed.ResultsWe found a considerable improvement in anesthesiology residents’ procedural skill performance, especially in the anesthesiology residency curriculum’s basic items. Besides, anesthesiology residents’ satisfaction was significantly improved after the 2nd DOPS.ConclusionsR-DOPS leads to improved training outcomes, including assessing the procedural skills, time to feedback to trainees, and trainee satisfaction.

16937 Evaluation of construct validity and inter-rater reliability of the Objective Structured Assessment of Technical Skills (OSATS) tool for assessment of surgical skills of dermatology trainees

Background: Assessment of procedural skills is an integral component of residency training. The OSATS tool, comprised of the global rating scale (GRS) and a suture-task checklist (STC), has been validated to evaluate surgical performance among surgical trainees. We assessed construct validity and inter-rater reliability of the OSATS tool in evaluating video recordings of dermatology residents throughout different stages of training.

The assessment of procedural skills in physiotherapy education: a measurement study using the Rasch model

BackgroundProcedural skills are a key element in the training of future physiotherapists. Procedural skills relate to the acquisition of appropriate motor skills, which allow the safe application of clinical procedures to patients. In order to evaluate procedural skills in physiotherapy education validated assessment instruments are required. Recently the assessment of procedural skills in physiotherapy education (APSPT) tool was developed. The overall aim of this study was to establish the structural validity of the APSPT. In order to do this the following objectives were examined: i) the fit of the items of APSPT to the Rasch-model, ii) the fit of the overall score to the Rasch model, iii) the difficulty of each test item and iv) whether the difficulty levels of the individual test items cover the whole capacity spectrum of students in pre-registration physiotherapy education.MethodsFor this observational cross-sectional measurement properties study a convenience sample of 69 undergraduate pre-registration physiotherapy students of the HES-SO Valais-Wallis was recruited. Participants were instructed to perform a task procedure on a simulated patient. The performance was evaluated with the APSPT. A conditional maximum likelihood approach was used to estimate the parameters of a partial credit model for polytomous item responses. Item fit, ordering of thresholds, targeting and goodness of fit to the Rasch model was assessed.ResultsItem fit statistics showed that 25 items of the APSPT showed adequate fit to the Rasch model. Disordering of item thresholds did not occur and the targeting of the APSPT was adequate to measure the abilities of the included participants. Undimensionality and subgroup homogeneity were confirmed.ConclusionThis study presented evidence for the structural validity of the APSPT. Undimensionality of the APSPT was confirmed and therefore presents evidence that the latent dimension of procedural skills in physiotherapy education consists of several subcategories. However, the results should be interpreted with caution given the small sample size.

Evaluating Resident Procedural Skills: Faculty Assess a Scoring Tool.

Procedural skills assessment is critical in residency training. The Council of Academic Family Medicine recommends the Procedural Competency Assessment Tool (PCAT) for assessing procedure competence of family medicine residents. We sought to evaluate the reliability of the PCAT and to better identify its strengths and limitations. In this mixed-methods study conducted in 2017, 18 faculty members of an academic family medicine residency program watched a video of one of the authors performing a simulated shave biopsy with intentional errors. Faculty scored the procedure using the shave biopsy PCAT, then participated in a focus group discussion of the rationale for the scores given. Qualitative analysis assessed perceived benefits and challenges of the PCAT. Following the discussion, faculty scored the same procedure again, using a PCAT modified with additional objective criteria. On the original PCAT, 40% of respondents rated the physician as competent. This dropped to 21.4% on the modified PCAT (P=.035). Respondents scored competent even though procedure components were scored as novice. Score variability decreased with the checklist-based PCAT. Qualitative analysis revealed that the PCAT is subjective and interpretation of the tool varies widely. Further studies regarding PCAT validity and reliability are needed. The PCAT may require further norming with additional objective criteria to improve reliability. Residencies may train faculty on using the PCAT to improve interobserver agreement, or decide to use a more intuitive checklist evaluation tool.

Procedural Training and Assessment for Pediatric Emergency Medicine Physicians Within the United States and Canada: A Survey Study.

The aims of this study were to determine current practices in procedural training and skill assessment for attending physicians working in pediatric emergency departments within the United States and Canada and identify barriers to providing training and assessment. This was a cross-sectional survey study. Members of the pediatric emergency medicine fellowship program directors and associate program directors Listserv were invited to participate in an anonymous survey about attending physician training and assessment practices for 9 specific procedures and barriers to training and assessment. Eighty-two (56.2%) of 146 recipients responded, with 79 surveys fully completed; 58.5% of responders report that their division offers procedural training, whereas 14.6% report assessment of procedural skills. The most common procedure for which participants report training and assessment is orotracheal intubation (53.1% and 7.5%, respectively), with training rates for other procedures ranging from 2.5% to 43.0%. Most sites that report training use simulation in some form for education. For assessment, simulation is used almost exclusively. Cost (50.6%), lack of faculty interest (36.7%), and lack of standardized guidelines (36.7%) are the most common barriers to training. Lack of standardized guidelines (51.9%), cost (43.0%), and lack of faculty interest (38.0%) are the most common barriers for assessment. Although pediatric emergency medicine physicians may be required to perform emergent procedures, opportunities to receive training and assessment in these procedures are limited. Simulation and other educational modalities are being used to provide skill training and assessment, but cost and lack of resources, standardized protocols, and faculty interest are barriers to the implementation of training and assessment programs.

Assessment methods in respiratory medicine training in Europe: current status and future needs.

There is a perceived need for harmonisation of training standards in medicine across Europe. Assessment methodology is a fundamental part of this harmonisation. Assessment may be: summative, measuring competency at the end of training, usually in a pass/fail mode; or formative, assessing strengths and weaknesses during training with a view to optimising performance. A survey of experienced respiratory medicine trainers in Europe was undertaken to discover the formative and summative assessment methods being used, and to explore the use of workplace-based assessments (WBAs). Structured interviews were sought with experienced trainers in adult and paediatric respiratory medicine. 35 trainers from 22 (71%) out of 31 countries were interviewed.The number of types of summative assessments required at the end of training varied from 0 to 4 including clinical (25%), written (44%) and oral (47%) examinations. Four respondents required a research thesis as a proof of clinical competence.WBA was not commonly used. Only 14% of respondents reported using a formal case-based discussion, 20% used a form of multisource feedback and 25% described some form of formal assessment of procedural skills. However, 77% of all respondents expressed a wish to have access to case-based discussion and 72% wanted to have a facility for multisource feedback. The majority also wanted training in the use of these tools. Almost half of the respondents had received no formal training in educational supervision and 80% of all respondents expressed a wish to receive such training.The findings suggest that there is no adequate process of ensuring uniform standards for specialist accreditation in Europe, and demonstrate a need and desire among trainers for more WBA tools and training in their use to be made available.

Characterizing the Development of Colonoscopy Competence Using Assessment Data.

Assessment of endoscopist competence is an increasingly important component of colonoscopy quality assurance. In this study from the Joint Advisory Group on Gastrointestinal Endoscopy, validity evidence is provided for the use of the Direct Observation of Procedural Skills assessment tool in the formative setting during training. In this national UK dataset, overall colonoscopy competence was typically achieved after 200-249 procedures, although certain complex procedural skills ("proactive problem solving" and "loop management") had not reached the threshold for competence even after 300 procedures. These data will help inform the development and/or refinement of certification policies and practices in jurisdictions around the world.

Development and Evaluation of a Proficiency-based and Simulation-based Surgical Skills Training for Technical Medicine Students.

This article was migrated. The article was marked as recommended. Objective Surgical graduate training to achieve practice-ready students is needed, yet is often lacking. This study developed and evaluated a proficiency-based, simulation-based course for basic surgical skills at graduate level. Learning outcomes were measured at the level of knowledge and skills and evaluated with a post-course questionnaire after students' clinical rotations. Methods The surgical skills course was anchored to surgical patient flow and covered topics and skills related to pre-, intra-, and post-operative care, including case-based medical reasoning, patient safety, infection management, operating theatre etiquette, scrubbing and donning, instrument handling, local anaesthesia, excision of tissue, and suturing. Students were assessed on knowledge and procedural skills. Results 155 graduate Technical Medicine students from academic years 2015-2016 and 2016-2017 entered this 10-week, 3 ECTS credits graduate Surgical Skills course. Pass rates of the knowledge test were 78%, and 87% for the procedural skill assessment. Graduate students reached proficiency level in a simulation-based basic surgical skills course. Students stated to go with confidence to the operating room and felt competent in performing four basic surgical skills. Conclusion Based on this study, we recommend that proficiency-based training using simulation should be standard in surgical curricula before students are allowed to practice on patients.

Colonoscopy Direct Observation of Procedural Skills Assessment Tool for Evaluating Competency Development During Training.

Formative colonoscopy direct observation of procedural skills (DOPS) assessments were updated in 2016 and incorporated into UK training but lack validity evidence. We aimed to appraise the validity of DOPS assessments, benchmark performance, and evaluate competency development during training in diagnostic colonoscopy. This prospective national study identified colonoscopy DOPS submitted over an 18-month period to the UK training e-portfolio. Generalizability analyses were conducted to evaluate internal structure validity and reliability. Benchmarking was performed using receiver operator characteristic analyses. Learning curves for DOPS items and domains were studied, and multivariable analyses were performed to identify predictors of DOPS competency. Across 279 training units, 10,749 DOPS submitted for 1,199 trainees were analyzed. The acceptable reliability threshold (G > 0.70) was achieved with 3 assessors performing 2 DOPS each. DOPS competency rates correlated with the unassisted caecal intubation rate (rho 0.404, P < 0.001). Demonstrating competency in 90% of assessed items provided optimal sensitivity (90.2%) and specificity (87.2%) for benchmarking overall DOPS competence. This threshold was attained in the following order: "preprocedure" (50-99 procedures), "endoscopic nontechnical skills" and "postprocedure" (150-199), "management" (200-249), and "procedure" (250-299) domain. At item level, competency in "proactive problem solving" (rho 0.787) and "loop management" (rho 0.780) correlated strongest with the overall DOPS rating (P < 0.001) and was the last to develop. Lifetime procedure count, DOPS count, trainer specialty, easier case difficulty, and higher cecal intubation rate were significant multivariable predictors of DOPS competence. This study establishes milestones for competency acquisition during colonoscopy training and provides novel validity and reliability evidence to support colonoscopy DOPS as a competency assessment tool.

Evaluation of Gender Differences in Ultrasound Milestone Evaluations During Emergency Medicine Residency Training: A Multicenter Study.

Prior literature has demonstrated incongruities among faculty evaluation of male and female residents' procedural competency during residency training. There are no known studies investigating gender differences in the assessment of procedural skills among emergency medicine (EM) residents, such as those required by ultrasound. The objective of this study was to determine if there are significant gender differences in ultrasound milestone evaluations during EM residency training. We used a stratified, random cluster sample of Accreditation Council for Graduate Medical Education (ACGME) EM residency programs to conduct a longitudinal, retrospective cohort analysis of resident ultrasound milestone evaluation data. Milestone evaluation data were collected from a total of 16 ACGME-accredited EM residency programs representing a 4-year period. We stratified milestone data by resident gender, date of evaluation, resident postgraduate year, and cohort (residents with the same starting date). A total of 2,554 ultrasound milestone evaluations were collected from 1,187 EM residents (750 men [62.8%] and 444 women [37.1%]) by 104 faculty members during the study period. There was no significant overall difference in mean milestone score between female and male residents [mean difference= 0.01 (95% confidence interval {CI}= -0.04 to 0.05)]. There were no significant differences between female and male residents' mean milestone scores at the first (baseline) PGY1 evaluation (mean difference= -0.04 [95% CI= -0.09 to 0.003)] or at the final evaluation during PGY3 (mean difference= 0.02 [95% CI= -0.03 to 0.06)]. Despite prior studies suggesting gender bias in the evaluation of procedural competency during residency training, our study indicates that there were no significant gender-related differences in the ultrasound milestone evaluations among EM residents within training programs throughout the United States.

Clinical procedural skills assessment during internship in ophthalmology

Introduction:Directly observed procedural skills (DOPS) is a unique method for assessment since it tests the trainee’s ability to apply his knowledge and skills in performing a particular procedure and provides an assessment of the practical work performed by the trainee on a ‘real’ patient under supervision of an experienced faculty. The study aims to make use of DOPS rating for assessment and further improvement in procedural skills in interns in Ophthalmology rotational posting. Methods: A prospective study was planned and 15 interns on 2 weeks’ rotational posting in the department of Ophthalmology were included by purposive sampling over a duration of 2 months. Four clinical procedural skills were identified and in the second week of posting, the interns were assessed by three DOPS encounters at an interval of 2 days for each clinical skill. The DOPS ratings were analyzed quantitatively using R-statistical software by repeated measure ANOVA and Banfuroni test. Results: A total of 180 DOPS were undertaken for 15 interns in 4 core areas of ophthalmic examination. The mean overall DOPS rating for DOPS-1 was 3.70±0.82, DOPS-2, 3.83±1.82 and DOPS-3, 4.93±1.65; the difference in DOPS rating between the first and second encounter was not statistically significant (p=0.497), between the second and the third and between the first and the third were statistically significant (p=0.000 in both cases) using Banfuroni test. The overall difference was also statistically significant (p=0.000) using repeated measure ANOVA. Both the assessor and intern satisfaction increased significantly from the first to the third DOPS, but not statistically significant between the first and the second and between the second and the third DOPS. Conclusions: We found significant improvement in interns’ clinical skills through repeated DOPS and the method was well accepted by both the students and the faculty. Internship period can be well utilized for improving clinical skills and novel performance assessment methods like DOPS might prove to be highly beneficial in ensuring adequacy of learning during internship and also to assess their readiness for accepting professional responsibilities in future.

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

BackgroundValidated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS.MethodsThis was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence.ResultsIn total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing ‘pre-procedure’, ‘technical’, and ‘post-procedure non-technical’ skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of ‘pre-procedure’ (100–124 procedures), ‘technical’ (150–174 procedures), ‘post-procedure non-technical’ skills (200–224 procedures), and global competency (225–249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence.ConclusionThis study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.

Identification of gender differences in ultrasound milestone assessments during emergency medicine residency training: a pilot study.

Objectives: Prior literature suggests that incongruities between male and female resident’s procedural competency may be explained by gender bias during the evaluation process. There are no known studies investigating gender differences in the assessment of ultrasound-based procedural skills among emergency medicine (EM) residents. The purpose of this study was to evaluate for gender differences in ultrasound milestone assessments among EM residents.Methods: This is a retrospective study including EM residents. Milestone assessment data were collected from a total of 3 Accreditation Council for Graduate Medical Education (ACGME) EM residency programs representing a 3-year period The outcome measures included mean milestone levels, milestone levels at baseline and graduation and differences in milestone achievement between female and male EM residents. An unpaired Student’s t-test was used to compare milestone scores between female and male residents.Results: A total of 456 ultrasound milestone evaluations were collected from 91 EM residents (34 females [37%] and 57 males [63%]). No significant differences were noted in the overall mean milestone level between females (2.3±0.6) and males (2.2±0.6) (P=0.387). There were no significant differences noted in the ultrasound milestone level between females (0.8±0.6) and males (0.7±0.7) at baseline (P=0.754). Although it did not reach statistical significance (P=0.197), the increase in the mean ultrasound milestone level from baseline to graduation was greater in males (3.4±0.7) compared to females (3.1±0.7).Conclusion: Overall, there were no statistically significant differences in the mean ultrasound milestone levels between females and males. The rate of ultrasound milestone level achievement during EM residency training at our institution had a slight tendency to be higher for males than females in the observed residency programs; however, this also did not reach statistical significance. Possible gender bias while evaluating ultrasound milestone levels needs to be further studied on a larger scale.

Direct observation of procedural skills (DOPS) evaluation method: Systematic review of evidence.

Background: Evaluation is one of the most important aspects of medical education. Thus, new methods of effective evaluation are required in this area, and direct observation of procedural skills (DOPS) is one of these methods. This study was conducted to systematically review the evidence involved in this type of assessment to allow the effective use of this method. Methods: Data were collected searching such keywords as evaluation, assessment, medical education, and direct observation of procedural skills (DOPS) on Google Scholar, PubMed, Science Direct, SID, Medlib and Google and by searching unpublished sources (Gray literature) and selected references (reference of reference). Results: Of 236 papers, 28 were studied. Satisfaction with DOPS method was found to be moderate. The major strengths of this evaluation method are as follow: providing feedback to the participants and promoting independence and practical skills during assessment. However, stressful evaluation, time limitation for participants, and bias between assessors are the main drawbacks of this method. Positive impact of DOPS method on improving student performance has been noted in most studies. The results showed that the validity and reliability of DOPS are relatively acceptable. Performance of participants using DOPS was relatively satisfactory. However, not providing necessary trainings on how to take DOPS test, not providing essential feedback to participants, and insufficient time for the test are the major drawbacks of the DOPS tests. Conclusion: According to the results of this study, DOPS tests can be applied as a valuable and effective evaluation method in medical education. However, more attention should be paid to the quality of these tests.