Quality Of Medical Education Research Research Articles

What is the learning effect of video review in postgraduate medical education: a systematic review

BackgroundVideo review is a feasible, commonly used learning tool, but current literature lacks a comprehensive review of its impact on learning in postgraduate medical education. This systematic review aims at examining the learning effect of video review of resident performance in clinical practice during postgraduate medical education.MethodsA systematic literature search was conducted from May 2023 to July 2023 with an update on 12/12/2023. Databases of MEDLINE (Pubmed), Web of Science, Embase and ERIC (through Webquest) were searched. Eligible articles had to describe the learning effects of video review in clinical practice in postgraduate medical education. The videos had to be actively recorded in a setting where a camera was not normally used for standard patient care. The investigated effect needed to be classified at least as a Kirkpatrick level 2. We iteratively developed a standardized data extraction form to extract study characteristics. The methodological quality of the individual studies was assessed using the Medical Education Research Quality Instrument.ResultsOut of 9323 records after deduplication, 11 studies were included. The designs were randomised controlled trials (n = 4) and single-group pre-test post-tests trials (n = 7). The studies had outcomes related to knowledge and skills (n = 5), resident behaviours (n = 5) and patient outcome (n = 1). All studies reported outcomes regarding learning effect.ConclusionsVideo review appears to have a positive impact on residents’ learning outcomes in postgraduate medical education. However, it is mostly not tailored to the specific learning needs of residents, and there is a lack of information regarding its optimal integration with other learning methods and within distinct clinical contexts. The heterogeneity observed among the included studies makes it challenging to formulate clear recommendations in the use of video.

Validity evidence for endoscopic ultrasound competency assessment tools: Systematic review.

Background and study aims Competent endoscopic ultrasound (EUS) performance requires a combination of technical, cognitive, and non-technical skills. Direct observation assessment tools can be employed to enhance learning and ascertain clinical competence; however, there is a need to systematically evaluate validity evidence supporting their use. We aimed to evaluate the validity evidence of competency assessment tools for EUS and examine their educational utility. Methods We systematically searched five databases and gray literature for studies investigating EUS competency assessment tools from inception to May 2023. Data on validity evidence across five domains (content, response process, internal structure, relations to other variables, and consequences) were extracted and graded (maximum score 15). We evaluated educational utility using the Accreditation Council for Graduate Medical Education framework and methodological quality using the Medical Education Research Quality Instrument (MERSQI). Results From 2081 records, we identified five EUS assessment tools from 10 studies. All tools are formative assessments intended to guide learning, with four employed in clinical settings. Validity evidence scores ranged from 3 to 12. The EUS and ERCP Skills Assessment Tool (TEESAT), Global Assessment of Performance and Skills in EUS (GAPS-EUS), and the EUS Assessment Tool (EUSAT) had the strongest validity evidence with scores of 12, 10, and 10, respectively. Overall educational utility was high given ease of tool use. MERSQI scores ranged from 9.5 to 12 (maximum score 13.5). Conclusions The TEESAT, GAPS-EUS, and EUSAT demonstrate strong validity evidence for formative assessment of EUS and are easily implemented in educational settings to monitor progress and support learning.

Leveraging evaluation of quality on medical education research with ChatGPT

What is the educational challenge? The Medical Education Research Study Quality Instrument (MERSQI) is widely used to evaluate the quality of quantitative research in medical education. It has strong evidence of validity and is endorsed by guidelines. However, the manual appraisal process is time-consuming and resource-intensive, highlighting the need for more efficient methods. What are the proposed solutions? We propose to use ChatGPT to evaluate the quality of medical education research with the MERSQI and compare its scoring with those of human evaluators. What are the potential benefits to a broader global audience? Using ChatGPT to evaluate medical education research with the MERSQI can decrease the resources required for quality appraisal. This allows faster summaries of evidence, reducing the workload of researchers, editors, and educators. Furthermore, ChatGPTs’ capability to extract supporting excerpts provides transparency and may have the potential for data extraction and training new medical education researchers. What are the next steps? We plan to continue evaluating medical education research with ChatGPT using the MERSQI and other instruments to determine its feasibility in this realm. Moreover, we plan to investigate which types of studies ChatGPT performs best in.

A78 VALIDITY EVIDENCE FOR OBSERVATIONAL EUS COMPETENCY ASSESSMENT: A SYSTEMATIC REVIEW

Abstract Background Endoscopic ultrasound (EUS) encompasses a range of diagnostic and therapeutic procedures that require technical, cognitive, and non-technical skills. The implementation of competency-based frameworks in endoscopic education has emphasized trainee assessment based on predefined milestones, rather than procedure volume. Observational assessment tools with strong validity evidence are needed to achieve this goal. Aims To systematically identify and evaluate observational competency assessment tools employed in EUS using an established validity framework. The secondary aim is to evaluate the educational utility of assessment tools. Methods We searched three databases (MEDLINE, EMBASE, and Evidence-Based Medicine Reviews) and the grey literature from inception to May 2023. Messick’s unified framework was used to evaluate validity evidence based on content, response process, internal structure, relations to other variables, and consequences. Each domain was scored from 0 to 3 with a maximum score of 15 points. Educational utility was evaluated using the Accreditation Council for Graduate Medical Education Standards considering ease of use, ease of interpretation, resources required, and educational impact. Study quality was assessed using the Medical Education Research Quality Instrument. Results Our search identified 2081 records. We screened 44 full texts and included 5 observational EUS assessment tools from 10 studies. All 5 tools are formative assessments, with 4 employed in clinical settings and one in a simulated setting. All tools use Likert rating scales and are rater-based, with 2 having additional self-assessment components. Validity evidence scores ranged from 3 to 13, with the EUS Assessment Tool (EUSAT), Global Assessment of Performance and Skills in EUS (GAPS-EUS), and The EUS and ERCP Skills Assessment Tool (TEESAT) scoring highest, with 10, 11, and 13 points, respectively. Overall educational utility was high across studies given ease of tool use. The TEESAT had the strongest educational impact considering its influence on credentialing and competence thresholds. Study quality was high overall, with scores ranging from 9.5 to 12 (maximum 13.5 points). Inter-rater agreement for validity evidence and educational utility scoring was substantial (k=0.73, raw agreement 80%) and almost perfect (k=0.92, raw agreement 96%), respectively. Conclusions The EUSAT, GAPS-EUS, and TEESAT demonstrate the strongest validity evidence for observational competency assessment of EUS and are easy to implement in educational settings. Future work should investigate barriers to implementation and evaluate utility of these tools for summative assessment. EUS Observational Competency Assessment Tool Validity Evidence Scores Funding Agencies None

Medical education research study quality instrument: an objective instrument susceptible to subjectivity

ABSTRACT Background: The medical education research study quality instrument (MERSQI) was designed to appraise medical education research quality based on study design criteria. As with many such tools, application of the results may have unintended consequences. This study applied the MERSQI to published medical education research identified in a bibliometric analysis. Methods: A bibliometric analysis identified highly cited articles in medical education that two authors independently evaluated using the MERSQI. After screening duplicate or non-research articles, the authors reviewed 21 articles with the quality instrument. Initially, five articles were reviewed independently and results were compared to ensure agreed upon understanding of the instrument items. The remainder of the articles were independently reviewed. Overall scores for the articles were analyzed with a paired samples t-test and individual item ratings were analyzed for inter-rater reliability. Results: There was a significant difference in mean MERSQI score between reviewers. Inter-rater reliability for MERSQI items labeled response rate, validity and outcomes were considered unacceptable. Conclusions: Based on these results there is evidence that MERSQI items can be significantly influenced by interpretation, which lead to a difference in scoring. The MERSQI is a useful guide for identifying research methodologies. However, it should not be used to make judgments on the overall quality of medical education research methodology in its current format. The authors make specific recommendations for how the instrument could be revised for greater clarity and accuracy.

A Systematic Review of Immersive Virtual Reality for Nontechnical Skills Training in Surgery

Immersive virtual reality (IVR) can be utilized to provide low cost and easily accessible simulation on all aspects of surgical education. In addition to technical skills training in surgery, IVR simulation has been utilized for nontechnical skills training in domains such as clinical decision-making and pre-operative planning. This systematic review examines the current literature on the effectiveness of IVR for nontechnical skill acquisition in surgical education. A literature search was performed using MEDLINE, EMBASE, and Web of Science for primary studies published between January 1, 1995 and February 9, 2022. Four reviewers screened titles, abstracts, full texts, extracted data, and analyzed included studies to answer 5 key questions: How is IVR being utilized in nontechnical skills surgical education? What is the methodological quality of studies? What technologies are being utilized? What metrics are reported? What are the findings of these studies? The literature search yielded 2340 citations, with 12 articles included for qualitative synthesis. Of included articles, 33% focused on clinical decision-making and 67% on anatomy/pre-operative planning. Motion sickness was a recorded metric in 25% of studies, with an aggregate incidence of 13% (11/87). An application score was reported in 33% and time to completion in 16.7%. A commercially developed application was utilized in 25%, while 75% employed a noncommercial application. The Oculus Rift was used in 41.7% of studies, HTC Vive in 25%, Samsung Gear in 16.7% of studies, Google Daydream in 8%, and 1 study did not report. The mean Medical Education Research Quality Instrument (MERSQI) score was 10.3 ± 2.3 (out of 18). In all studies researching clinical decision-making, participants preferred IVR to conventional teaching methods and in a nonrandomized control study it was found to be more effective. Averaged across all studies, mean scores were 4.33 for enjoyment, 4.16 for utility, 4.11 for usability, and 3.73 for immersion on a 5-point Likert scale. The IVR nontechnical skills applications for surgical education are designed for clinical decision-making or anatomy/pre-operative planning. These applications are primarily noncommercially produced and rely upon a diverse array of HMDs for content delivery, suggesting that development is primarily coming from within academia and still without clarity on optimal utilization of the technology. Excitingly, users find these applications to be immersive, enjoyable, usable, and of utility in learning. Although a few studies suggest that IVR is additive or superior to conventional teaching or imaging methods, the data is mixed and derived from studies with weak design. Motion sickness with IVR remains a complication of IVR use needing further study to determine the cause and means of mitigation.

Undergraduate teaching of surgical skills in the UK: systematic review.

Students must be proficient in surgical skills according to General Medical Council and Royal College of Surgeons of England guidelines. If these skills are not appropriately taught, there is a risk of an incoming junior workforce with inadequate surgical skills. This paper aimed to review the literature relating to undergraduate teaching of surgical skills in the UK and summarize future suggested training methods. The databases MEDLINE, Embase and SCOPUS were searched, and the existing literature relating to methodology of undergraduate teaching of surgical skills in the UK over the past 10 years was summarized. The Medical Education Research Quality Instrument was used to assess research quality. A total of 19 papers were included. Cross-sectional evaluations and survey-based studies highlight a clear deficit in surgical skills teaching in the UK. Medical students are currently unable to fulfil their own learning needs and meet requirements set out by the General Medical Council. This lack of surgical teaching appears to negatively affect student desire to pursue a surgical career. The three main themes for improvement are extracurricular surgical skills days, near-peer teaching and simulation. Each method appeared to improve learning, although no studies utilized medium- to long-term follow-up to demonstrate efficacy and there lacks a clear consensus as to the 'standard' of undergraduate surgical skill education. There was also potential for selection bias and response shift bias in many of the studies assessing pre- and postintervention confidence and opinions. There is a concerning lack of surgical skills teaching that has resulted in medical students and junior doctors not having the necessary surgical skills as per General Medical Council guidance and students feel that their own learning needs are not met. This failure to address the learning deficit may be responsible for the fall in surgical competition ratios. While surgical skills teaching must be improved urgently, more robust evidence is required to evaluate the optimal ways of approaching this issue.

Quality of Evidence and Pedagogical Strategy in Using Gamification in Medical Education Literature: A Systematic Review

Introduction Gamification is the use of game principles in non-game settings such as medical education. Effective evidence for using gamification in medical education needs well-designed studies that describe outcomes related to interventions. The study objective was to systematically review the study designs describing gamification interventions in medical education focusing on their characteristics and medical students’ learning. Methods We systematically searched the databases, including Google Scholar, PubMed, Scopus, and Web of Science until 2021. Articles were appraised using the Medical Education Research Quality Instrument (MERSQI). Results The search result included 423 articles, of which 53 articles had inclusion criteria. Of these, 23 articles were classified in the non-electronic games group and 30 studies in the electronic games group. The quality of evidence in both groups was assessed using the MERSQI average checklist. In total, 90% of studies reported the effectiveness of game intervention in one of the 4 levels of the Kirkpatrick pyramid. The predominant pedagogical approach in both groups was the cognitive approach. The psychomotor approach and the affective approach were more prominent in the electronic and non-electronic games groups, respectively. Conclusion This study shows that gamification training techniques improve the performance of medical students in most cases.

Validity evidence for observational ERCP competency assessment tools: a systematic review.

BACKGROUND : Assessment of competence in endoscopic retrograde cholangiopancreatography (ERCP) is critical for supporting learning and documenting attainment of skill. Validity evidence supporting ERCP observational assessment tools has not been systematically evaluated. METHODS : We conducted a systematic search using electronic databases and hand-searching from inception until August 2021 for studies evaluating observational assessment tools of ERCP performance. We used a unified validity framework to characterize validity evidence from five sources: content, response process, internal structure, relations to other variables, and consequences. Each domain was assigned a score of 0-3 (maximum score 15). We assessed educational utility and methodological quality using the Accreditation Council for Graduate Medical Education framework and the Medical Education Research Quality Instrument, respectively. RESULTS : From 2769 records, we included 17 studies evaluating 7 assessment tools. Five tools were studied for clinical ERCP, one for simulated ERCP, and one for simulated and clinical ERCP. Validity evidence scores ranged from 2 to 12. The Bethesda ERCP Skills Assessment Tool (BESAT), ERCP Direct Observation of Procedural Skills Tool (ERCP DOPS), and The Endoscopic Ultrasound (EUS) and ERCP Skills Assessment Tool (TEESAT) had the strongest validity evidence, with scores of 10, 12, and 11, respectively. Regarding educational utility, most tools were easy to use and interpret, and required minimal additional resources. Overall methodological quality (maximum score 13.5) was strong, with scores ranging from 10 to 12.5. CONCLUSIONS : The BESAT, ERCP DOPS, and TEESAT had strong validity evidence compared with other assessments. Integrating tools into training may help drive learners' development and support competency decision making.

A117 VALIDITY EVIDENCE FOR ENDOSCOPIC RETROGRADE CHOLANGIOPANCREATOGRAPHY COMPETENCY ASSESSMENT TOOLS: A SYSTEMATIC REVIEW

Abstract Background Assessment of competence in endoscopic retrograde cholangiopancreatography (ERCP) is essential to ensure trainees possess the skills needed for independent practice. Traditionally, ERCP training has used the apprenticeship model, whereby novices learn skills under the supervision of an expert. A growing focus on procedural quality, however, has supported the implementation of competency-based medical education models which require documentation of a trainee’s competence for independent practice. Observational assessment tools with strong evidence of validity are critical to this process. Validity evidence supporting ERCP observational assessment tools has not been systematically evaluated. Purpose To conduct a systematic review of ERCP assessment tools and identify tools with strong evidence of validity using a unified validity evidence framework Method We conducted a systematic search using electronic databases and hand-searching from inception until August 2021 for studies evaluating observational assessment tools of ERCP performance. We used a unified validity framework to characterize validity evidence from five sources: content, response process, internal structure, relations to other variables, and consequences. Each domain was assigned a score of 0-3 (maximum score 15). We assessed educational utility and methodological quality using the Accreditation Council for Graduate Medical Education framework and the Medical Education Research Quality Instrument, respectively. Result(s) From 2769 records, we included 17 studies evaluating 7 assessment tools. Five tools were studied for clinical ERCP, one on simulated ERCP, and one on simulated and clinical ERCP. Validity evidence scores ranged from 2-12. The Bethesda ERCP Skills Assessment Tool (BESAT), ERCP Direct Observation of Procedural Skills Tool (ERCP DOPS), and The Endoscopic Ultrasound (EUS) and ERCP Skills Assessment Tool (TEESAT) had the strongest validity evidence with scores of 10, 12, and 11, respectively. Regarding educational utility, most tools were easy to use and interpret, and required minimal additional resources. Overall methodological quality was strong, with scores ranging from 10-12.5 (maximum 13.5). Conclusion(s) The BESAT, ERCP DOPS, and TEESAT have strong validity evidence compared to other assessments. Integrating tools into training may help drive learners’ development and support competency decision-making. Please acknowledge all funding agencies by checking the applicable boxes below CAG Disclosure of Interest None Declared

Assessing changes in the quality of quantitative health educations research: a perspective from communities of practice

BackgroundAs a community of practice (CoP), medical education depends on its research literature to communicate new knowledge, examine alternative perspectives, and share methodological innovations. As a key route of communication, the medical education CoP must be concerned about the rigor and validity of its research literature, but prior studies have suggested the need to improve medical education research quality. Of concern in the present study is the question of how responsive the medical education research literature is to changes in the CoP. We examine the nature and extent of changes in the quality of medical education research over a decade, using a widely cited study of research quality in the medical education research literature as a benchmark to compare more recent quality indicators.MethodsA bibliometric analysis was conducted to examine the methodologic quality of quantitative medical education research studies published in 13 selected journals from September 2013 to December 2014. Quality scores were calculated for 482 medical education studies using a 10-item Medical Education Research Study Quality Instrument (MERSQI) that has demonstrated strong validity evidence. These data were compared with data from the original study for the same journals in the period September 2002 to December 2003. Eleven investigators representing 6 academic medical centers reviewed and scored the research studies that met inclusion and exclusion criteria. Primary outcome measures include MERSQI quality indicators for 6 domains: study design, sampling, type of data, validity, data analysis, and outcomes.ResultsThere were statistically significant improvements in four sub-domain measures: study design, type of data, validity and outcomes. There were no changes in sampling quality or the appropriateness of data analysis methods. There was a small but significant increase in the use of patient outcomes in these studies.ConclusionsOverall, we judge this as equivocal evidence for the responsiveness of the research literature to changes in the medical education CoP. This study identified areas of strength as well as opportunities for continued development of medical education research.

Fundus-first laparoscopic cholecystectomy for complex gallbladders: A systematic review

Retrograde ‘fundus-first’ cholecystectomy (FF) signifies the dissection that starts from the fundus of the gallbladder to the infundibulum in case structures of Calot's triangle cannot be identified. Although feasible in laparoscopic cholecystectomy (LC), FF remains an underutilized approach in difficult cases. We aimed to systematically review the fundus-first laparoscopic cholecystectomy (FFLC) and to evidence-base its advantages and feasibility. A systematic review was performed in compliance with PRISMA guidelines. A literature search was performed using PubMed/MEDLINE, ScienceDirect and Cochrane-Library for articles published from 2001 to 2021. Search keywords included ‘retrograde cholecystectomy’, ‘fundus-first cholecystectomy’ and ‘fundus-down cholecystectomy’. Quality assessments were applied using the Medical Education Research Quality Instrument (MERSQI) scores. Also, evidence levels were employed using GRADE. The protocol was registered with PROSPERO register (CRD42021227518). Altogether 9393 citations were identified and reviewed for this study. A final 23 studies were included, with a total of 7973 cholecystectomies comprising 3020 with FF approach. The endpoints were operative time, duration of postoperative hospital-stay and intraoperative and postoperative complications, as well as rate of conversion to open surgery. MERSQI mean score was 10.2 (SD= 1.85). The FF dissection was evidenced to be a superior technique when compared to conventional anterograde dissection as regards duration of operation, pain, nausea, conversion to open surgery and duration of sick leave. Furthermore, FF was found to be appropriate for difficult LC The fundus-first laparoscopic cholecystectomy was associated with a shorter operating time, decrease in pain and nausea scores and reduced incidence of conversion to open cholecystectomy. Ultrasonic dissection was favoured in the retrograde dissection compared to that with electrocautery.

Training and assessment of musculoskeletal ultrasound and injection skills-a systematic review.

To examine how residents are trained and assessed in musculoskeletal US (MSUS), MSUS-guided and landmark-guided joint aspiration and injection. Additionally, to present the available assessment tools and examine their supporting validity evidence. A systematic search of PubMed, Cochrane Library and Embase was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and studies published from 1 January 2000 to 31 May 2021 were included. Two independent reviewers performed the search and data extraction. The studies were evaluated using the Medical Education Research Quality Instrument (MERSQI). A total of 9884 articles were screened, and 43 were included; 3 were randomized studies, 21 pre- and post-test studies, 16 descriptive studies and 3 studies developing assessment tools. The studies used various theoretical training modalities, e.g. lectures, anatomical quizzes and e-learning. The practical training models varied from mannequins and cadavers to healthy volunteers and patients. The majority of studies used subjective 'comfort level' as assessment, others used practical examination and/or theoretical examination. All training programs increased trainees' self-confidence, theoretical knowledge, and/or practical performance, however few used validated assessment tools to measure the effect. Only one study met the MERSQI high methodical quality cut-off score of 14. The included studies were heterogeneous, and most were of poor methodological quality and not based on contemporary educational theories. This review highlights the need for educational studies using validated theoretical and practical assessment tools to ensure optimal MSUS training and assessment in rheumatology.

Social Media in Health Studies: A Systematic Review of Comparative Learning Methods.

Learning modifications particularly increased due to the SARS-CoV-2 pandemic, which made it necessary to offer distance health education for many months. Social media allows students to have interactive activities such as discussing specific subjects or sharing data with each other, and also to have interactions with their teachers and tutors. So, we aimed to compare the effects of the use of social media on knowledge, skills and perceptions in health students compared to other methods. We performed a systematic review on PubMed, ScienceDirect and Embase about comparative learning methods using social media. The search followed PRISMA guidelines, and the quality assessment of the studies was performed using the Medical Education Research Quality instrument (MERSQI). Eight studies were analyzed including 1014 participants. Mean age ranged from 19.9 to 23.4 years, and 70% were females. About 54.4% of the participants were medical students and 20.9% were dental students. The mean MERSQI was 11.7 ± 2.6. Various subjects were included—anatomy, cultural competences, sterile surgical techniques, radiology, arthrocentesis, medical pathologies and cariology. As far as knowledge evaluation was concerned, we found that the use of social media may have had a positive effect from a short-term point of view but results concerning skills were less consistent across studies. Students usually had a positive perception of the use of social media as a complementary method but not as a complete alternative so it is not excluded that this effect might result from an increase in working time. The impact on patient care should also be assessed in future studies.

SP6.1.8Fundus-first laparoscopic cholecystectomy for complex gallbladders: A systematic review

Abstract Aims Retrograde ‘fundus-first’ cholecystectomy is when the dissection starts from the fundus of the gallbladder to the infundibulum, in case structures of Calot's triangle cannot be identified. Although feasible in laparoscopic cholecystectomy, it remains an underutilized approach. We aimed to systematically review the fundus-first laparoscopic cholecystectomy and study its safety and feasibility. Methods A systematic review was performed in compliance with PRISMA guidelines. A literature search was performed using PubMed/MEDLINE, ScienceDirect and Cochrane-Library for articles published from 2010 to 2020. Search keywords included ‘retrograde cholecystectomy’, ‘fundus-first cholecystectomy’ and ‘fundus-down cholecystectomy’. Quality assessments were applied using the Medical Education Research Quality Instrument (MERSQI) score. The protocol was registered with PROSPERO register. Results A total of 3503 studies formed the base for evidence evaluations. 12 studies with 1978 fundus-first cholecystectomies were assessed. Three citations were scored high and 5 moderate quality. Endpoints included blood loss, rate of conversion to open, bile duct injury, gallbladders perforations, postoperative pain and hospital stay. Nine studies provided both strong and moderate scientific evidence for a positive outcome of the fundus-first approach. Conclusion The fundus-first cholecystectomy was associated with a reduced need for intraoperative cholangiography, shorter operating time, lower incidence of intra-operative complications, bile duct injury and reduced incidence of conversion to open cholecystectomy.

Colonoscopy competence assessment tools: a systematic review of validity evidence.

Assessment tools are essential for endoscopy training, being required to support feedback provision, optimize learner capabilities, and document competence. We aimed to evaluate the strength of validity evidence that supports the available colonoscopy direct observation assessment tools using the unified framework of validity. We systematically searched five databases for studies investigating colonoscopy direct observation assessment tools from inception until 8 April 2020. We extracted data outlining validity evidence (content, response process, internal structure, relations to other variables, and consequences) from the five sources and graded the degree of evidence, with a maximum score of 15. We assessed educational utility using an Accreditation Council for Graduate Medical Education framework and methodological quality using the Medical Education Research Quality Instrument (MERSQI). From 10 841 records, we identified 27 studies representing 13 assessment tools (10 adult, 2 pediatric, 1 both). All tools assessed technical skills, while 10 each assessed cognitive and integrative skills. Validity evidence scores ranged from 1-15. The Assessment of Competency in Endoscopy (ACE) tool, the Direct Observation of Procedural Skills (DOPS) tool, and the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) had the strongest validity evidence, with scores of 13, 15, and 14, respectively. Most tools were easy to use and interpret, and required minimal resources. MERSQI scores ranged from 9.5-11.5 (maximum score 14.5). The ACE, DOPS, and GiECAT have strong validity evidence compared with other assessments. Future studies should identify barriers to widespread implementation and report on the use of these tools in credentialing examinations.

A97 TOOLS FOR DIRECT OBSERVATION AND ASSESSMENT OF COLONOSCOPY: A SYSTEMATIC REVIEW OF VALIDITY EVIDENCE

Abstract Background An increasing focus on quality and safety in colonoscopy has led to broader implementation of competency-based educational systems that enable documentation of trainees’ achievement of the knowledge, skills, and attitudes needed for independent practice. The meaningful assessment of competence in colonoscopy is critical to this process. While there are many published tools that assess competence in performing colonoscopy, there is a wide range of underlying validity evidence. Tools with strong evidence of validity are required to support feedback provision, optimize learner capabilities, and document competence. Aims We aimed to evaluate the strength of validity evidence that supports available colonoscopy direct observation assessment tools using the unified framework of validity. Methods We systematically searched five databases for studies investigating colonoscopy direct observation assessment tools from inception until April 8, 2020. We extracted data outlining validity evidence from the five sources (content, response process, internal structure, relations to other variables, and consequences) and graded the degree of evidence, with a maximum score of 15. We assessed educational utility using an Accreditation Council for Graduate Medical Education framework and methodological quality using the Medical Education Research Quality Instrument (MERSQI). Results From 10,841 records, we identified 27 studies representing 13 assessment tools (10 adult, 2 pediatric, 1 both). All tools assessed technical skills, while 10 assessed cognitive and integrative skills. Validity evidence scores ranged from 1–15. The Assessment of Competency in Endoscopy (ACE) tool, the Direct Observation of Procedural Skills (DOPS) tool, and the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT) had the strongest validity evidence, with scores of 13, 15, and 14, respectively. Most tools were easy to use and interpret and required minimal resources. MERSQI scores ranged from 9.5–11.5 (maximum score 14.5). Conclusions The ACE, DOPS, and GiECAT have strong validity evidence compared to other assessments. Future studies should identify barriers to widespread implementation and report on use of these tools in credentialing purposes. Funding Agencies None

Effect of mental rehearsal on team performance and non-technical skills in surgical teams: systematic review.

BackgroundSimulation‐based training in medical education has become a common method to develop both technical and non‐technical skills in teams. Mental rehearsal (MR) is the cognitive act of simulating a task in our heads to pre‐experience tasks imaginatively. It has been used widely to improve individual and collective performance in fields outside healthcare, and offers potential for more efficient training in time‐pressured surgical and medical team contexts. This study aimed to review the available literature to determine the impact of MR on team performance and non‐technical skills in healthcare.MethodsMEDLINE, Embase, British Educational Index, CINAHL, Web of Science, PsycInfo and Cochrane databases were searched for the period 1994–2018. The primary outcome measure was improvement in team performance and non‐technical skills. Study quality of RCTs was assessed using the Medical Education Research Quality Instrument. The reported impacts of MR in all included studies were mapped on to the Kirkpatrick framework for evaluation of educational interventions.ResultsEight studies with 268 participants were identified that met the inclusion criteria, of which there were six randomized trials, one prospective pragmatic trial and one qualitative study. Three studies found MR to be effective in improving team non‐technical skills. MR practices were varied and often poorly defined. MR benefited team non‐technical skills when it was specifically designed to do so, but was not an automatic consequence of technical MR alone. The majority of studies demonstrated benefits of MR for technical performance, but only three showed positive impacts on teamwork. Overall the studies were of low quality and lacked sufficient discriminatory focus to examine impacts on teamwork dynamics.ConclusionMR can improve technical performance, but the benefits on non‐technical skills are less clear. Future research should look at longitudinal mixed‐method evaluation designs and focus on real clinical teams.

Contemporary practice of standardised bedside teaching rounds.

SUMMARYBackgroundThe purpose of this article is to review the extant literature on bedside teaching rounds within the context of ward rounds performed with the entire clinical team at the patient’s bedside, and to assess the effects of standardisation of this process on patient and learner satisfaction, as well as other duty‐hour restrictions and patient care metrics in the academic inpatient setting. Ultimately, the intent of this review is to inform faculty development sessions for educators on the benefits and challenges of standardised rounding protocols.MethodsWe performed a search of PubMed, Scopus and CINAHL databases (from 2003 to August 2019). Randomised, controlled trials, pre‐ and post‐interventional studies and cohort studies, in English, were eligible for inclusion. Two reviewers independently searched, screened and analysed the studies, and a narrative synthesis was performed. Articles were evaluated methodologically using the Medical Education Research Quality Study Instrument (MERQSI).ResultsFive articles were included, with one randomised controlled trial, three cohort studies, and one pre‐ and post‐interventional study. The collective MERSQI score for the studies was 12.3. Patient satisfaction increased uniformly across studies when standardised practices were used. Attempts to improve learner satisfaction, however, achieved mixed results. In addition, the time of bedside rounds was found to decrease with standardised interventions overall.ConclusionIn light of generally positive albeit limited evidence for standardised rounding practices, faculty development initiatives might use these data to inform and educate faculty members regarding the use of standardised protocols for bedside rounds.

Clay-Based Modeling in the Anatomist's Toolkit: A Systematic Review.

Anatomical education has suffered from reduced teaching time and poor availability of staff and resources over the past thirty years. Clay-based modeling (CBM) is an alternative technique for teaching anatomy that can improve student knowledge and experience. This systematic review aimed to summarize and appraise the quality of the literature describing the uses, advantages, and limitations of CBM compared to alternative methods of teaching human gross anatomy to students or qualified healthcare professionals. A systematic search of Embase, MEDLINE, Scopus, and Web of Science databases was conducted, and the Medical Education Research Quality Instrument (MERSQI) was used to assess study quality. Out of the 829 studies identified, 12 papers met the inclusion criteria and were eligible for this review. The studies were of high quality, with a mean MERSQI score of 11.50/18. Clay-based modeling can be used to teach all gross anatomical regions, and 11 studies demonstrated a significant improvement in short-term knowledge gain in students who used CBM in comparison to other methods of learning anatomy. Eight studies that included subjective assessment showed that CBM is rated highly. However, some studies showed that students viewed CBM as juvenile and experienced difficulty making the models. Additionally, there is no evidence to suggest that CBM improves long-term knowledge. Clay-based modeling is an effective learning method for human gross anatomy and should be incorporated into the anatomists' toolkit. In the future, more randomized controlled studies with transparent study designs investigating the long-term impact of CBM are needed.