Direct Observation Of Procedural Skills Research Articles

Non‐technical skills curriculum incorporating simulation‐based training improves performance in colonoscopy among novice endoscopists: Randomized controlled trial

Non-technical skills (NTS), involving cognitive, social and interpersonal skills that complement technical skills, are important for the completion of safe and efficient procedures. We investigated the impact of a simulation-based curriculum with dedicated NTS training on novice endoscopists' performance of clinical colonoscopies. A single-blinded randomized controlled trial was conducted at a single center. Novice endoscopists were randomized to a control curriculum or a NTS curriculum. The control curriculum involved a didactic session, virtual reality (VR) simulator colonoscopy training, and integrated scenario practice using a VR simulator, a standardized patient, and endoscopy nurse. Feedback and training were provided by experienced endoscopists. The NTS curriculum group received similar training that included a small-group session on NTS, feedback targeting NTS, and access to a self-reflective NTS checklist. The primary outcome was performance during two clinical colonoscopies, assessed using the Joint Advisory Group Direct Observation of Procedural Skills (JAG DOPS) tool. Thirty-nine participants completed the study. The NTS group (n = 21) had superior clinical performance during their first (P < 0.001) and second clinical colonoscopies (P < .0.001), compared to the control group (n = 18). The NTS group performed significantly better on the VR simulator (P < 0.05) and in the integrated scenario (P < 0.05). Our findings demonstrate that dedicated NTS training led to improved performance of clinical colonoscopies among novices.

Assessment of suitability of direct observation of procedural skills among postgraduate students and faculty in periodontology and implantology

Purpose: Direct observation of procedural skills (DOPS) is widely used on the basis of workplace-based assessments, which consist of direct observation and feedback. The DOPS covers all related professional qualities including knowledge, clinical perception, communication skills, ethics, the rights of patients, and the speed and accuracy of the task. However, the use of DOPS in dental treatment, especially in the field of periodontics is limited. Objective: The aim of the study was to find the utility of DOPS as assessment tool for the periodontal procedure. Methodology: The departmental faculty and postgraduate (PG) students were sensitized to the DOPS concept. A total of 12 PG students performed the periodontal procedure and were assessed on the basis of the DOPS rating scale for the 4 encounters each. The student's clinical skills were assessed by faculty members and then both students and faculty members were asked to give feedback for the procedure and overall experience. Results: Faculty and students both were comfortable in giving the feedback. Improvements in the knowledge of different clinical skills like relevant anatomy, technique of procedure were observed from 1st encounter to 4th encounter from 70% to 73.33%. Taking informed consent was improved from 53.33% to 90% from 1st encounter to 4th. Moreover, it helped the student to develop and improve their clinical skills, patient examination, and decision-making and treatment. Conclusion: The present study finding shows that faculty and students both were comfortable in giving the feedback and it helped the student to develop and improve their clinical skills. Our experience indicates that DOPS is an appropriate and functional tool in PG setup to enhance the clinical competencies.

The effect of direct observation of procedural skills method on learning clinical skills of midwifery students of medical sciences.

INTRODUCTION:Clinical education is one of the most important parts of medical students' education, and it is a major part of the education of qualified and professional people. Therefore, this study was conducted to determine the effect of applying Direct Observation of Procedural Skills (DOPS) on midwifery students' clinical skills.MATERIALS AND METHODS:This is a quasi-experimental, two-group study conducted as a pre- and post-study on midwifery students in 2017–2018. Cluster and randomized sampling method was used. The processors involved in this study were three main skills of vaginal examination, pelvic examination, and vaginal delivery. The DOPS method was used to assess the practical skills in the interventional group during three times the process at day, 1 day, and at least 1 week later; the usual logbook method was used in the control group. Two groups were evaluated at the end of the midwifery course by Comprehensive Final Midwifery checklist. The tools were checked for validity and reliability, and data were analyzed using descriptive and analytical statistics.RESULTS:There was no statistically significant difference between the two groups in terms of important demographic variables such as age, grade, marital status, and initial assessment score (P > 0.05). The mean of final scores in the normal delivery, vaginal examination, and pelvimetry was statistically significantly higher in the interventional group (P < 0.001). On the other hand, the functional field of the students in the interventional group was statistically significantly improved in normal delivery and pelvimetry (P < 0.05), and this difference was not significant in the vaginal examination. In addition, the mean scores of students before and after the DOPS method were statistically significantly different in every skill in Comprehensive Final Midwifery checklist (P < 0/05).CONCLUSIONS:The DOPS assessment methodology is not only a useful tool of clinical evaluation, but also an effective tool for clinical learning of students. For this purpose, it is suggested that educational members of midwifery take enough time to design DOPS method in the same process.

Formative assessment in practical for Indian postgraduates in health professions education: A strategic initiative towards competency-based education.

Background:Observational assessment (OA) or Workplace Based Assessment (WPBA) is being increasingly used to assess the trainees by direct observation to shape their learning. The current deficiencies in our conventional assessment system are due to lack of conceptualization of assessment as a process for continuous improvement and learning, leading to non-utilization of many available tools of assessment.Objective:The present study aimed to introduce formative assessment (FA) for postgraduates at all the constituent colleges (Medical, Dental, Ayurveda, Nursing) under the ambit of Datta Meghe Institute of Medical Sciences (Deemed University) Sawangi (M) Wardha, (Maharashtra), India.Methodology:Observational Assessment tools used for FA were Mini CEX (Mini Clinical Evaluation Exercise) and DOPS (Direct Observation of Procedural Skills)in addition to Objectively Structured Clinical/Practical Examination (OSCE/OSPE). Six encounters of OAs were conducted at the end of each semester. The OA tool used was either the MiniCEX or/and DOPS, depending on the type of subject (Preclinical, Para-clinical, Clinical).Results:A significant improvement in scores from one semester to other as the postgraduate learner progresses, was observed. The overall response from the postgraduate students and faculty for FA was satisfactory.Conclusion:Formative assessment in practical's, was feasible, acceptable and effective program to improve the overall learning and competency of postgraduates.

Effectiveness of simulation based teaching of ventilatory management among non-anaesthesiology residents to manage COVID 19 pandemic - A Quasi experimental cross sectional pilot study.

Background and Aims:Simulation is one of the important learning tools when it comes to skill acquisition and as a supplemental tool for training in high stake situations like COVID-19. The aim of this study is to meet the global requirements of knowledge on ventilatory management, prepare and to evaluate the effectiveness of the teaching module for non-anesthesiology trainees on COVID-19 patients.Methods:Quasi experimental cross sectional pilot study was conducted with a sample of twenty-six trainees. A teaching module was prepared and validated which consisted of lectures, audio-video sessions, demonstrations with hands-on training, debriefing, analytical-phase and reflection. Pre and Post evaluations from student t-test and direct observation of procedural skills (DOPS) were used for knowledge and skill assessment respectively and feedback obtained from Likert's score.Results:Pre- and Post-tests had a mean score of 7.42 ± 2.12 and 14.92 ± 2.9 respectively (P value 0.00001). DOPS included 16 point score, in which 23 trainees (88.4%) met the expectations and above expectations as per training objectives. A five-point Likert's score feedback revealed satisfactory and highly satisfactory scores of 100% (ABG), 96.1% (mechanical ventilation), and 84.6% (ventilation in COVID-19 patients). Overall satisfaction for the workshop among respondents was 100 per cent. Confidences of handling scores were 84.5% (interpreting ABG), 65.3% (maneuvering mechanical ventilation), and 96.15% (intubation in COVID-19 patients).Conclusion:A planned teaching module in ventilation management helps to train non-anaesthesiologists more effectively as a part of COVID-19 preparedness. Simulation with debriefing based training to the medical fraternity is the best alternative in the present pandemic and it will also ensure the safety of health care professionals.

Work-Based Assessments in Higher General Surgical Training Program: A Mixed Methods Study Exploring Trainers' and Trainees' Views and Experiences.

Introduction In the United Kingdom, work-based assessments (WBAs) including procedure-based assessments (PBAs), case-based discussions (CBDs), clinical evaluation exercises (CEXs), and direct observation of procedural skills (DOPS) have been used in Higher General Surgical Training Program (HGSTP) since the introduction of Modernising Medical Careers. Although the Intercollegiate Surgical Curriculum Project states that they should be used for the formative development of trainees using feedback and reflection, there is no study to look at the perception of their usefulness and barriers in using them, particularly in HGSTP. The aim of this study is to investigate trainer's and trainee's perception of their usefulness, barriers in using them, and way forward for their improvement in HGSTP. Methods This was a mixed method study. In phase I, after ethics committee approval, an online survey was sent to 83 trainers and 104 trainees, with a response rate of 33 and 37%, respectively, using Online Surveys (formerly Bristol Online Survey) from July 2018 to December 2018. After analysis of this result, in phase II, semistructured interviews were conducted with five trainees and five trainers who had volunteered to take part from phase I. Thematic analysis was performed to develop overarching themes. Results For professional formative development, 15% of the trainers and 53% of the trainees felt that WBAs had a low value. Among 4 WBAs—CEX, CBD, PBA, and DOPS—PBA was thought to be the most useful WBA by 52% trainers and 74% trainees. More trainers than trainees felt that it was being used as a formative tool (33 vs. 16%). The total number of WBAs thought to be required was between 20 and 40 per year, with 46% of the trainers and 53% of the trainees preferring these numbers.The thematic analysis generated four themes with subthemes in each: theme 1, “factors affecting usefulness,” including the mode of validation, trainer/trainee engagement, and time spent in validating; theme 2, “doubt on utility” due to doubt on validity and being used as a tick-box exercise; theme 3, “pitfalls/difficulties” due to lack of time to validate, late validation, e-mail rather than face-to-face validation, trainer and trainee behavior, variability in feedback given, and emphasis on number than quality; and theme 4, “improvement strategies.” Conclusions The WBAs are not being used in a way they are supposed to be used. The perception of educational impact (Kirkpatrick levels 1 and 2) by trainers was more optimistic than by trainees. Improvements can be made by giving/finding more time, trainer training, more face-to-face validation, and better trainer trainee interactions.

Internal Assessment in New MBBS Curriculum: Methods and Logistics.

Assessment is a pivotal element of competency-based curriculum. The implementation of competency-based undergraduate medical curriculum in India requires proper implementation of updates in assessment for which universities, colleges, and teachers need to plan and design internal assessment (IA) modules as well as guidelines. IA provides opportunity to assess many competencies and hence should be implemented judiciously. Multiple assessment methods should be used to improve utility of IA. The process should involve all the teachers of a subject and all competencies. Capacity building trainings should be organized by institutes in basic concepts of assessment as well as training in methods such as the objective structured clinical/practical examination, direct observation of procedural skills, and mini-clinical evaluation exercises. The culture of providing regular feedback needs to be instilled at institute levels. The learners who are not able to achieve competencies and required criteria in university examination should be provided predecided remedial measures for improving their performance. The article discusses all these aspects in detail.

Validating written feedback in clinical formative assessment

Formative assessment is widely accepted as being crucial to promoting student learning and, since 2010, the UK General Medical Council has mandated its use in workplace-based clinical training for all new doctors. As a result, the Royal College of Radiologists (RCR) instituted a range of formative workplace-based assessments including the Radiology Direct Observation of Procedural Skills (Rad-DOPS), in which supervisors appraise trainees’ performance in carrying out clinical procedures. This paper reports on the quality of the written feedback in 2,500 Rad-DOPS online feedback forms in addressing the aims of the new assessment approach. Random samples of 500 were selected from the first three years of the new assessment implementation, 2010–13, and from 2016 to 17. Using an appropriate coding frame, the feedback was analysed across the samples against key trainee attributes including stage of training and level of adjudged competence. Criteria for identifying high quality feedback were derived from the literature and a simplified form of qualitative comparative analysis was used to identify the conditions associated with high quality feedback. An average of 97% of the assessments contained written feedback but the number of instances of high quality feedback was found to be exceedingly small at around 5%. The paper offers suggestions for making the feedback process more purposeful in achieving the aims of formative assessment.

Are workplace-based assessment methods (DOPS and Mini-CEX) effective in nursing students’ clinical skills? A single-blind randomized, parallel group, controlled trial

Background: Evaluation of clinical skills is critically important for nursing students. However, the quality of evaluation tools is poor.Objectives: To evaluate the effectiveness of Direct Observation of Procedural Skills (DOPS) and Mini-Clinical Evaluation Exercise (Mini-CEX) on clinical skills of nursing students.Methods: This study was conducted among 108 senior nursing students. Mini-CEX and DOPS were utilized to evaluate clinical skills in the intervention group.Results: The mean of students’ scores in all of the five procedures was significantly higher in the intervention group compared to control group.. Students’ scores for the procedures significantly raised through the first stage of DOPS and Mini-CEX to the third stage.Conclusions: Utilization of DOPS and Mini-CEX for evaluation of clinical skills in nursing students effectively enhance their learning ability. Implementing of such assessment methods lead to promoting clinical skills of students which eventually help them to provide high quality care for their patients.

ERCP assessment tool: evidence of validity and competency development during training.

The endoscopic retrograde cholangiopancreatography (ERCP) direct observation of procedural skills (DOPS) is a 27-item competency assessment tool that was developed to support UK ERCP training. We evaluated validity of ERCP DOPS and competency development during training. This prospective study analyzed ERCP DOPS performed in the UK between July 2016 and October 2018. Reliability was measured using Cronbach's alpha, and DOPS scores were benchmarked using the contrasting groups method. The percentage of competent scores was averaged for each item, domain, and overall rating, and stratified by lifetime procedure count to evaluate learning curves. Multivariable analyses were performed to identify predictors of DOPS competence. 818 DOPS (109 trainees, 80 UK centers) were analyzed. Overall Cronbach's alpha was 0.961. Attaining competency in 87 % of assessed DOPS items provided the optimal competency benchmark. This was achieved in the domain sequence of: pre-procedure, post-procedure management, endoscopic non-technical skills, cannulation & imaging, and execution of selected therapy, and across all items after 200 - 249 procedures (89 %). After 300 procedures, the benchmark was reached for selective cannulation (89 %), but not for stenting (plastic 73 %; metal 70 %), sphincterotomy (80 %), and sphincteroplasty (56 %). On multivariable analysis, lifetime procedure count (P = 0.002), easier case difficulty (P < 0.001), trainee grade (P = 0.03), and higher lifetime DOPS count (P = 0.01) were predictors of DOPS competence. This study provides novel validity, reliability, and learning curve data for ERCP DOPS. Trainees should have a minimum of 300 hands-on ERCP procedures before undertaking summative assessment for independent practice.

Undergraduate ultrasound education at German-speaking medical faculties: a survey.

Background: The purpose of this study was twofold: to assess the status of undergraduate medical ultrasound (US) education in the German-speaking area and to suggest a possible framework for a longitudinal undergraduate medical US curriculum based on the study results and a literature review.Methods: The survey included 44 medical faculties in the German-speaking area: 37 in Germany, four in Austria and three in German-speaking Switzerland. A standardized questionnaire focused on the following aspects of undergraduate medical US education: general information, organization, resources, assessment methods and evaluation. Results: Data from 28 medical faculties were analysed. 26 out of 28 medical faculties offered US courses, 21 offered compulsory as well as elective courses, four offered compulsory and one elective courses only. 27 medical faculties supported US skills implementation. Abdominal US (n=25) was most common in teaching basic US skills. A learning objective catalogue was provided at 15 medical faculties. At 22 medical faculties, medical specialists were involved in undergraduate medical US education. 24 out of 26 medical faculties thought that peer-teaching is important to convey US skills. Medical faculties used the following methods to assess US skills: objective structured clinical examination (OSCE, n=7), non-standardized practical exams (n=4), non-standardized combined oral-practical exams (n=2) or direct observation of procedural skills (DOPS, n=1). 25 out of 26 medical faculties evaluated their US courses and 19 made suggestions for improvements in undergraduate medical US education.Conclusion: Medical faculty members in the German-speaking area have recognized the relevance of undergraduate medical US education. So far, courses are offered heterogeneously with rather short hands-on scanning time and high student-instructor ratio. Based on the results of this study and a literature review we suggest a possible framework and milestones on the way to a longitudinal undergraduate medical US curriculum.

Paediatric Colonoscopy Direct Observation of Procedural Skills: Evidence of Validity and Competency Development.

The paediatric series of direct observation of procedural skills (DOPS) were introduced into the UK national endoscopy training curriculum in 2016, but lack validity evidence. We aimed to present validity evidence for paediatric colonoscopy DOPS and study competency development in a national trainee cohort. This prospective UK-wide study analysed formative paediatric colonoscopy DOPS which were submitted to the e-Portfolio between 2016 and 2018. Item, domain, and average DOPS scores were correlated with the overall DOPS rating to evidence internal structure validity. Overall DOPS ratings were compared over lifetime procedure count to demonstrate learning curves (discriminant validity). Consequential validity was founded on receiver operating characteristic curve analyses. A total of 203 DOPS assessments were completed for 29 trainees from 11 UK training centres. Internal structure validity was provided through item-total correlation analyses. DOPS scores positively correlated with trainee seniority (P < 0.001) and lifetime procedure count (P < 0.001). Competency acquisition followed the order of: "preprocedure," "postprocedure," "endoscopic nontechnical skills," "management," "procedure" domains, followed by overall DOPS competency, which was achieved in 81% of the cohort after 125 to 149 procedures. Mean DOPS scores could be used to predict overall procedure competence (area under receiver operating characteristic curve 0.969, P < 0.001), with a mean score of 3.9 demonstrating optimal sensitivity (93.5%) and specificity (87.6%). This study provides validity evidence supporting the use of paediatric colonoscopy DOPS as an in-training competence assessment tool. DOPS may also be used to measure competency development and benchmark performance during training, which may be of value to trainees, trainers, and training programmes.

Deliberations on the contemporary assessment system

Deliberations on the contemporary assessment system

Workplace based assessment by Direct Observation of procedural skills for postgraduate students in Obstetrics and Gynaecology

Introduction: Structured workplace based assessment in PG curriculum is the need of the hour. This project addresses the same. This study was conducted to assess procedural skill of postgraduate students by using a novel tool-Direct Observation of Procedural skills (DOPS) and impact on learning by reassessment. Methodology: Seven second year postgraduate students and eight faculty members participated in the study. First DOPS cycle was conducted over 6 weeks for 5 basic procedures in Obstetrics and Gynecology (OBGY). The checklists of procedures were shared. Second DOPS cycle was conducted by same faculty. At the end, feedback about the entire process was collected and analyzed. Results: All the students showed improvement in DOPS scores during second session. The component task wise score analysis showed statistically significant improvement in following areas such as maintenance of asepsis, pre procedure preparations, demonstration of anatomy and indications, communication with patient, post procedure management and overall tissue handling. This was evident when ‘paired t test’ was applied to the data. Analysis of feedback showed acceptability, feasibility and relevance of DOPS in OBGY. Conclusion: We found that DOPS is highly acceptable assessment tool by both the postgraduate students and faculty. It helped students to identify gaps in learning and achieve improvement in skills.

Effectiveness of DOPS “direct observation of procedural skills” as a method of formative assessment for improving the clinical skills of post-graduate students in the department of obstetrics and gynecology

JETHS - J Educ Technol Health Sci- Print ISSN No:-2454-4396 Online ISSN No:-2393-8005,Journal of Education Technology in Health Sciences (JETHS) open access, peer-reviewed triannually journal publishing since 2014 and is published under auspices of the “NKP Salve Institute of Medical Sciences and Research Centre

Effectiveness of Direct Observation of Procedural Skills (DOPS) in Postgraduate Training in Urology at Institute of Kidney Diseases, Peshawar.

To determine the effectiveness of direct observation of procedural skills (DOPS) in the assessment and improving operative skills of postgraduate residents in urology. Comparative study. Department of Urology, Institute of Kidney Diseases (IKD), Hayatabad Medical Complex, Peshawar, from March till December 2016. Twenty postgraduate residents in urology were assessed for three commonly performed procedures in urology with structured proforma, specially designed for DOPS, using Likert scale from 1 to 5. Formative feedback was given to all the residents highlighting their strengths and weaknesses. During second phase, all the three procedures were performed by four consultants respectively with residents as learner in the second phase of DOPS. During the third phase, a repeat assessment of e-residents was performed to study the overall improvement in operative skills, respectively. In pre-DOPS assessment, only 7 (35%) out of 20 residents, passed in skill of cystoscopy, 6 (30%) residents passed in transurethral resection of prostate (TURP), while 3 (15%) residents passed in skills of uretero-renoscopy (URS). The mean score of overall ability to perform procedure during first phase of DOPS were 2.7 ±1.3 for cystoscopy, 2.1 ±0.97 for TURP, and 2.5 ±1.1 for ureterorenoscopy. Significant improvement (p 0.04) in passing was observed in third phase with 17 (85%), 14 (70%), and 14 (70%) residents were declared passed in cystoscopy, TURP and URS, respectively. The mean score of overall ability to perform procedure during the third phase was 4.7 ±0.7 for cystoscopy, 4.5 ±0.94 for TURP, and 4.7 ±0.94 for ureteroscopy. DOPS is effective tool for assessing and improving the skills of postgraduate residents in urology.

An exploration of the training and evaluation systems of medical undergraduates in the context of tutorial system

Objective To investigate the training process of clinical practice ability in medical undergraduates in the context of tutorial system, as well as new clinical training models and evaluation methods. Methods A total of 40 medical undergraduates who entered the stage of clinical practice were randomly divided into tutorial group and control group, with 20 undergraduates in each group. A diversified evaluation system was used to perform a comprehensive evaluation of clinical practice ability of medical undergraduates at different stages, with a combination of professional theory examination, Mini-Clinical Evaluation Exercise, Direct Observation of Procedural Skills, and clinical case report. All the data were analyzed by SPSS 23.0, t test was used for statistical analysis. Results Compared with the control group after the internship ended, the tutorial group had significantly higher scores of clinical professional theory(86.24±2.93), clinical practice ability (7.53±0.48), clinical operation skills (4.35±0.47), and clinical thinking ability (6.31±0.52) (all P<0.05). Conclusion The application of the tutorial training mode during the internship of medical undergraduates can improve their comprehensive qualities such as professional quality, medical theory, and practical skills. The diversified evaluation system can evaluate the teaching effect and reflect the cultivation of comprehensive quality among students. Key words: Tutorial system; Diversified evaluation system; Clinical practice ability

Effect of use of direct observation of procedural skills for assessment for learning in Obstetrics and Gynaecology postgraduate students at Medical University, Bhutan: a prospective study

Introduction: Direct Observation of Procedural Skills (DOPS) is a valuable method to evaluate procedural skills. The aim of this study was to explore the effect of DOPS as an assessment method and its effect on their learning in Obstetrics and Gynecology Postgraduate students. Methods: Five residents and four faculty members from the department of Obstetrics and Gynecology, Khesar Gyalpo University of Medical Sciences of Bhutan participated in this study. A validated checklist was adopted, from which five core areas of DOPS were utilized for this study. The two skills selected for this study were Pap smear and Intra Uterine Contraceptive Device (IUCD) insertion. The acquired data were analyzed by descriptive statistics (frequency, percent and mean). Results: The results showed that student’s initial mean score was 4/9, which increased to nearly 8/9 in third attempt. Practically, all faculty members and students believed that DOPS was effective in facilitating students’ learning. Some of the students thought that repeated exposure to DOPS reduced their level of stress. Conclusions: It may be stated that DOPS is effective in facilitating students’ learning and clinical skills. Keywords: Assessment; DOPS; learning; Obstetrics and Gynecology; WPBA.

Clinical procedural skills assessment during internship in ophthalmology

Introduction:Directly observed procedural skills (DOPS) is a unique method for assessment since it tests the trainee’s ability to apply his knowledge and skills in performing a particular procedure and provides an assessment of the practical work performed by the trainee on a ‘real’ patient under supervision of an experienced faculty. The study aims to make use of DOPS rating for assessment and further improvement in procedural skills in interns in Ophthalmology rotational posting. Methods: A prospective study was planned and 15 interns on 2 weeks’ rotational posting in the department of Ophthalmology were included by purposive sampling over a duration of 2 months. Four clinical procedural skills were identified and in the second week of posting, the interns were assessed by three DOPS encounters at an interval of 2 days for each clinical skill. The DOPS ratings were analyzed quantitatively using R-statistical software by repeated measure ANOVA and Banfuroni test. Results: A total of 180 DOPS were undertaken for 15 interns in 4 core areas of ophthalmic examination. The mean overall DOPS rating for DOPS-1 was 3.70±0.82, DOPS-2, 3.83±1.82 and DOPS-3, 4.93±1.65; the difference in DOPS rating between the first and second encounter was not statistically significant (p=0.497), between the second and the third and between the first and the third were statistically significant (p=0.000 in both cases) using Banfuroni test. The overall difference was also statistically significant (p=0.000) using repeated measure ANOVA. Both the assessor and intern satisfaction increased significantly from the first to the third DOPS, but not statistically significant between the first and the second and between the second and the third DOPS. Conclusions: We found significant improvement in interns’ clinical skills through repeated DOPS and the method was well accepted by both the students and the faculty. Internship period can be well utilized for improving clinical skills and novel performance assessment methods like DOPS might prove to be highly beneficial in ensuring adequacy of learning during internship and also to assess their readiness for accepting professional responsibilities in future.

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

BackgroundValidated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS.MethodsThis was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence.ResultsIn total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing ‘pre-procedure’, ‘technical’, and ‘post-procedure non-technical’ skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of ‘pre-procedure’ (100–124 procedures), ‘technical’ (150–174 procedures), ‘post-procedure non-technical’ skills (200–224 procedures), and global competency (225–249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence.ConclusionThis study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.