Assessment In Medical Education Research Articles

Abstract WP390: The UTHealth Stomp Out Stroke Festival- A City Wide Public Education Initiative

Introduction: TJC certification for comprehensive stroke centers requires two annual public education activities on stroke prevention. One of our stroke faculty designed a multidisciplinary and multicultural program to educate the region and involved local businesses by establishing the UTHealth Stomp Out Stroke Festival. Hypothesis: We analyzed the benefits to the community of sponsoring a city wide, multicultural stroke educational event Methods: Implementation included establishing leadership and health screening providers, engaging educators and community stakeholders, recruiting volunteers and healthcare workers from hospital affiliates, pharmacy, rehabilitation and the schools of public health, nursing, dentistry, and medicine and developing a marketing and communication plan. Three areas for the event were created for Stroke Prevention, Stroke Recovery and a Kids Zone. Results: 1st Festival was in 2013 with 100 participants, the 2nd in 2014 with 609 participants and the 3rd in 2015 with 1,261 participants. Over the 3 years, we have progressively increased the number of attendees, volunteers, health educators and community partners. Table 1 shows demographics for year 3. In year 3, we provided over 20,000 patient education materials, conducted 224 blood pressure screenings, completed 145 stroke risk scorecards, and provided FAST education to 109 children. Other assessments included BMI, cholesterol, bone density, fall risks, sleep apnea. We had 23 Chinese, 25 Spanish, and 10 Vietnamese health worker-translators. Conclusion: We implemented a city wide public health initiative providing medical education, health screenings, and risk assessments to a highly ethnically diverse population in Houston. Major challenges include funding, needing volunteers and translators, engaging media, and maximizing community participation. Next steps will focus on determining the effectiveness to improve health literacy and behavioral health changes.

Introducing a legal mandate for basic medical education accreditation in Korea.

Introducing a legal mandate for basic medical education accreditation in Korea.

Revisiting ‘Measuring the process of solving clinical diagnostic problems’

Revisiting ‘Measuring the process of solving clinical diagnostic problems’

A novel mentorship programme for residents integrating academic development, clinical teaching and graduate medical education assessment.

IntroductionMentorship fosters career development and growth. During residency training, mentorship should support clinical development along with intellectual and academic interests. Reported resident mentoring programmes do not typically include clinical components. We designed a programme that combines academic development with clinical feedback and assessment in a four-year emergency medicine residency programme.MethodsIncoming interns were assigned an advisor. At the conclusion of the intern year, residents actively participated in selecting a mentor for the duration of residency. The programme consisted of quarterly meetings, direct clinical observation and specific competency assessment, assistance with lecture preparation, real-time feedback on presentations, simulation coaching sessions, and discussions related to career development. Faculty participation was recognized as a valuable component of the annual review process. Residents were surveyed about the overall programme and individual components.ResultsOver 88 % of the respondents said that the programme was valuable and should be continued. Senior residents most valued the quarterly meetings and presentation help and feedback. Junior residents strongly valued the clinical observation and simulation sessions.ConclusionsA comprehensive mentorship programme integrating clinical, professional and academic development provides residents individualized feedback and coaching and is valued by trainees. Individualized assessment of clinical competencies can be conducted through such a programme.

Does Making the Numerical Values of Verbal Anchors on a Rating Scale Available to Examiners Inflate Scores on a Long Case Examination?

Rating scales are frequently used for scoring assessments in medical education. The effect of changing the structural elements of a rating scale on students' examination scores has received little attention in the medical education literature. This study assessed the impact of making the numerical values of verbal anchors on a rating scale available to examiners in a long case examination (LCE). During the 2011-2012 academic year, the numerical values of verbal anchors on a rating scale for an internal medicine clerkship LCE were made available to faculty examiners. Historically, and specifically in the control year of 2010-2011, examiners only saw the scale's verbal anchors and were blinded to the associated numerical values. To assess the impact of this change, the authors compared students' LCE scores between the two cohort years. To assess for differences between the two cohorts, they compared students' scores on other clerkship assessments, which remained the same between the two cohorts. From 2010-2011 (n = 226) to 2011-2012 (n = 218), the median LCE score increased significantly from 82.11% to 85.02% (P < .01). Students' performance on the other clerkship assessments was similar between cohorts. Providing examiners with the numerical values of verbal anchors on a rating scale, in addition to the verbal anchors themselves, led to a significant increase in students' scores on an internal medicine clerkship LCE. When constructing or changing rating scales, educators must consider the potential impact of the rating scale structure on students' scores.

Systematic Review of Computer Based Assessments in Medical Education.

Medical schools, postgraduate training institutes, licensing and certification bodies have developed and implemented many new methods for accurate, reliable, and timely assessments of the competence of medical professionals and practicing physicians. The underlying objective of all these assessments is to not only evaluate the students’ learning and educational goals but also to establish the graduating individual's skills and professionalism. Computer based assessment (CBA) has emerged in recent years as a viable alternative to traditional assessment techniques. It has also infiltrated and influenced the medical curriculum where it has been employed for assessment tasks. This study presents how CBA offers pedagogical opportunities and analyzes its usage pattern over the past three decades. We examined 47 CBAs in medical education and analyzed several assessment components, including application area, assessment purpose, assessment type, assessment format, student level, and emphasized the interplay among these components. Our analysis determined that formative assessment is the most frequently used type and 75% of all assessment types employed used the multiple choice questions format.

Development of a Method to Measure Clinical Reasoning in Pediatric Residents: The Pediatric Script Concordance Test

Introduction: The Script Concordance Test (SCT) is an assessment method of clinical reasoning skills. SCT is designed to assess a candidate’s ability to reason when faced with decisions encountered in the three phases of clinical decision-making: diagnosis, investigation and treatment. Challenges have been raised related to psychometric properties of SCT scores. Data about accept- ability of the SCT method are also needed. Objectives: 1) To examine the validity of a Pediatric Script Concordance Test (PSCT) in discriminating clinical reasoning ability between junior postgraduate year (PGY) 1 - 2 and senior PGY 3 - 4 pediatric residents, and pediatricians, 2) To determine if higher reliability could be achieved by applying specific test design strategies to the PSCT and 3) To explore trainees’/physicians’ acceptability of the PSCT. Methods: A 24-case/137 question PSCT was administered to 91 residents from four Canadian training centers. Each resident’s PSCT was scored based on the aggregate responses of 21 pediatricians (Panel of Experts (POE)). ANOVA was used to compare across the 3 levels of experience. Reliability was calculated using Cronbach’s α coefficient. Participants completed a post-test survey about the acceptability of PSCT. Results: Overall, a statistical difference in performance was noted across all levels of experience, F = 22.84 (df = 2); p α) was 0.85. Participants expressed keen interest and engagement in the PSCT. Conclusions: PSCT is a valid, reliable, feasible and acceptable method to assess the core competency of clinical reasoning. We suggest the PSCT may be effectively integrated into formative residency assessment and with increasing exposure, experience and refinement may soon be ready to pilot within summative assessments in pediatric medical education.

New HybridLab Training Method of Clinical Skills Learning in Neonatology: Students’ Opinion [Correction

MedEdPublish (ISSN 2312–7996) is a highly visible, open access, specialist practitioner e-journal that enables academics, teachers, clinicians, researchers and students to publish their experiences, views and research findings relating to teaching, learning and assessment in medical and health professions education. An innovative and key feature of MedEdPublish is that the peer review process begins following publication.

Педагогічне оцінювання у підготовці компетентного лікаря в Україні: проблеми застосування і перспективи вирішення

In article deals with the problem of the use of educational assessment in the training of future doctors in Ukraine. The key role of educational assessment in the improvement of the quality of higher medical education was defined. The basic stages of educational assessment methods in historical perspective are presented. Specifies, that educational assessment is one of the components of educational expertise. Analysis the use of the most common method of educational assessment – testing was found, that it plays an important but only supporting role in determining the level of formation of professional competence of future doctor. The successful experience of foreign countries indicates that for the increasing of objectiveness of the educational outcomes estimation need to get information from multiple information sources. Above all, to confirm the academic achievements of future doctors is necessary at first to base on their self-assessment. These data are supplemented with the preliminary assessment of teachers, the results of the test control and eventually should be interpreted and compared with current estimates in discipline. Such integrated approach helps identify problematic aspects of the future doctors training, find decisions to remove them and draft the perspectives of future learning outcomes. Besides, the development of tests should be based on the principles of competence approach, focusing on the mechanisms of multiple knowledge integration and practical orientation of them. The analysis of scientific sources and own teaching experience shows that the effectiveness of solving problems in educational assessment in high medical school, including dental education, largely depends on the creation of a comprehensive system of quality assessment of higher medical education at the national level, expert teams in each higher medical schools; implementation of integrated and differentiated approach into the process of educational assessment; training of experts in field of educational assessment. Decreasing of efficiency test method is seen in insufficient theoretical and methodological support of educational assessment process in general and testing process in particular.

Steps towards Updating the Curriculum and Teaching Methods in Obstetrics, Gynaecology and Neonatology in Mongolia

Background: Medical education in Mongolia faces many challenges in terms of staff capacity, large student numbers, and limited access to clinical experience. The Government of Mongolia has placed a high priority on reducing maternal and infant mortality, necessitating improvements to the quality of medical training, ensuring a highly skilled health workforce is produced and maintained. In 2014, a team of academic staff from Sydney Medical School were appointed by United Nations Population Fund (UNFPA) to assist in reviewing and updating the medical student curriculum in obstetrics, gynaecology and neonatology in accordance with international best practice. The first phase involved a visit from a senior delegation from Mongolia, including representatives from the Mongolian National University of Medical Sciences (MNUMS), UNFPA and the Mongolian Ministry of Health to Sydney. The week long programme was designed to demonstrate best practice in obstetrics, gynaecology and neonatology undergraduate medical education; and display modern teaching practices. Methods: Course design included demonstration and participation in a four station Structured, Clinical, Objective, Reference, problem-oriented, Integrated, and Organised (SCORPIO); observation of a Clinical Reasoning Session (CRS); demonstration and participation in a four station OSCE, and teaching of best practice in writing single-best answer multiple choice questions. Participants also took part in a teacher training session. The programme was implemented at a large teaching hospital in Sydney, Australia. We employed mixed-methods to evaluate the programme, using pre- and post-questionnaires and a focus group. Results: The programme increased participants’ perceived understanding and ability to apply educational principles, plan learning activities, and provide feedback. In particular, participants perceived that their understanding how to implement SCORPIO, CRS and OSCE had increased. However, participants would have liked greater opportunity to observe bedside teaching. Participants foresaw challenges to the implementation of educational changes in Mongolia, including the anticipated difficulty of engaging hospital staff in teaching; implementing a student-centred approach to teaching; and providing a large number of students with adequate clinical experience. Conclusion: Changes in educational strategy in Mongolia may assist medical schools to produce clinically competent graduates. Our programme provided an effective means to introduce Mongolian leaders in health and education to modern student-centred medical education teaching and assessment methods; and to highlight the importance of teacher training and evaluation as a strategy to engage both university and hospital staff in medical education. Additionally, programme outcomes assisted in subsequent phases of the project, including in-country needs assessment, curriculum development and delivery.

In the minds of OSCE examiners: uncovering hidden assumptions.

The Objective Structured Clinical Exam (OSCE) is a widely used method of assessment in medical education. Rater cognition has become an important area of inquiry in the medical education assessment literature generally, and in the OSCE literature specifically, because of concerns about potential compromises of validity. In this study, a novel approach to mixed methods that combined Ordinal Logistic Hierarchical Linear Modeling and cognitive interviews was used to gain insights about what examiners were thinking during an OSCE. This study is based on data from the 2010 to 2014 administrations of the Clinician Assessment for Practice Program OSCE for International Medical Graduates (IMGs) in Nova Scotia. An IMG is a physician trained outside of Canada who was a licensed practitioner in a different country. The quantitative data were examined alongside four follow-up cognitive interviews of examiners conducted after the 2014 administration. The quantitative results show that competencies of (1) Investigation and Management and (2) Counseling were highly predictive of the Overall Global score. These competencies were also described in the cognitive interviews as the most salient parts of OSCE. Examiners also found Communication Skills and Professional Behavior to be relevant but the quantitative results revealed these to be less predictive of the Overall Global score. The interviews also reveal that there is a tacit sequence by which IMGs are expected to proceed in an OSCE, starting with more basic competencies such as History Taking and building up to Investigation Management and Counseling. The combined results confirm that a hidden pattern exists with respect to how examiners rate candidates. This study has potential implications for research into rater cognition, and the design and scoring of practice-ready OSCEs.

Electronical educational and methodological complexes as a means of forming the professional competence of future doctors when studying disciplines from natural sciences

The article examines the possibilities of electronical educational and methodological complexes for the formation of professional competence of masters of higher medical education institutions and assessment of their academic achievements. It is claimed that obtaining a quality education is inextricably linked to the use of digital information and communication technologies, connected with the ability of students to understand electronic materials presented in the form of texts, graphs and tables, and to independently master theoretical material. Emphasis is placed on the tendency to use professionally oriented materials as a source of knowledge. The article presents the results of research into the forms and methods of effective implementation of electronical educational and methodical complexes on the Moodle platform among students which are studying at the Kharkiv National Medical University on the example of the academic discipline Medical and Biological Physics. The advantages and disadvantages of using electronic educational and methodological complexes in higher educational institutions are discussed.

Entrustable Professional Activity—Based Evaluations of Gastroenterology Fellows to Aid in Informing ACGME Reporting Milestones

Abstract Over the past 15 years, there has been a shift in medical trainee education assessment and evaluation. The landmark change included the now well-known six core competencies proposed by the...

Variation in assessment and standard setting practices across UK undergraduate medicine and the need for a benchmark.

ObjectivesThe principal aim of this study is to provide an account of variation in UK undergraduate medical assessment styles and corresponding standard setting approaches with a view to highlighting the importance of a UK national licensing exam in recognizing a common standard.MethodsUsing a secure online survey system, response data were collected during the period 13 - 30 January 2014 from selected specialists in medical education assessment, who served as representatives for their respective medical schools.ResultsAssessment styles and corresponding choices of standard setting methods vary markedly across UK medical schools. While there is considerable consensus on the application of compensatory approaches, individual schools display their own nuances through use of hybrid assessment and standard setting styles, uptake of less popular standard setting techniques and divided views on norm referencing.ConclusionsThe extent of variation in assessment and standard setting practices across UK medical schools validates the concern that there is a lack of evidence that UK medical students achieve a common standard on graduation. A national licensing exam is therefore a viable option for benchmarking the performance of all UK undergraduate medical students.

The Impact of item flaws, testing at low cognitive level, and low distractor functioning on multiple-choice question quality.

BackgroundThis study investigated the impact of addressing item writing flaws, testing at low cognitive level and non-functioning distractors (< 5 % selection frequency) in multiple-choice assessment in preclinical medical education.MethodMultiple-choice questions with too high or too low difficulty (difficulty index < 0.4 or > 0.8) and insufficient discriminatory ability (point-biserial correlation < 0.2) on previous administration were identified. Items in Experimental Subgroup A underwent removal of item writing flaws along with enhancement of tested cognitive level (21 multiple-choice questions), while Experimental Subgroup B underwent replacement or removal of non-functioning distractors (11 multiple-choice questions). A control group of items (Group C) did not undergo any intervention (23 multiple-choice questions).ResultPost-intervention, the average number of functioning distractors (≥ 5 % selection frequency) per multiple-choice question increased from 0.67 to 0.81 in Subgroup A and from 0.91 to 1.09 in Subgroup B; a statistically significant increase in the number of multiple-choice questions with sufficient point-biserial correlation was also noted. No significant changes were noted in psychometric characteristics of the control group of items.ConclusionCorrection of item flaws, removal or replacement of non-functioning distractors, and enhancement of tested cognitive level positively impact the discriminatory ability of multiple-choice questions. This helps prevent construct-irrelevant variance from affecting the evidence of validity of scores obtained in multiple-choice questions.Electronic supplementary materialThe online version of this article (doi:10.1007/s40037-015-0212-x) contains supplementary material, which is available to authorized users.

Reflections on the First 2 Years of Milestone Implementation.

The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) collectively constitute the foundation of professional self-regulation in the United States. In February 1999, the 2 organizations approved 6 general competencies broadly relevant for all medical practice, followed by the official launch of the Outcomes Project in 2001. It was expected that the competencies would be an antidote to overspecification of accreditation standards, and that they would empower programs to create training programs grounded in meaningful outcomes in a developmental approach. As many programs can attest, the implementation of outcomes-based (eg, competency-based) medical education has been challenging. One reason has been the difficulty in implementing the competencies in both curriculum and assessment. Program leaders lacked shared mental models within their own training programs, accompanied by a lack of shared understanding nationally within disciplines. It is important to remember that 1 of the thorny problems the milestones were intended to address was the sources of unwanted and unwarranted variability in educational and, by extension, clinical outcomes. In addition, the community cannot improve at scale what cannot be measured, and prior frames and approaches to measurement were insufficient and ineffective. A key goal for milestones thus is to help improve the state and quality of measurement through better assessment in graduate medical education to facilitate the improved outcomes everyone desires. Approximately 10 years ago, conversations began on how to more effectively and meaningfully operationalize the competencies to help improve the design of residency and fellowship programs through the use of a developmental framework. In parallel, the ACGME began to explore mechanisms to move the accreditation system to a focus on outcomes using a continuous quality improvement philosophy. Developmental milestones, using narratives to describe in more descriptive terms the professional trajectories of residents, were seen as a way to move the outcomes project forward. Starting in 2007, the disciplines of internal medicine, pediatrics, and surgery began to create developmental milestones for the 6 competencies. Surgery would subsequently delay the development of their milestones focusing first on the SCORE curriculum. The ACGME began to restructure its accreditation processes in 2009, and soon after, milestone groups were constituted for all specialties. Milestone writing groups were cosponsored by the ACGME and the ABMS member certification boards. Early groups had significant latitude in developing their subcompetencies and milestones; specialties that started the process after 2010 used a standard template. Each milestone set was subjected to review by the educational community in the specialty. BOX 1 provides an overview of the purposes of the milestones across key stakeholders, and FIGURE 1 provides an example of a key driver diagram of milestones as an educational and clinical intervention. As FIGURE 1 highlights, milestones can potentially trigger a number of drivers, or mechanisms, to help enable changes in residency and fellowship education. In 2013, the milestones were officially launched in 7 core specialties (emergency medicine, internal medicine, neurological surgery, orthopaedic surgery, pediatrics, diagnostic radiology, and urology) as a formative, continuous quality improvement component of the new accreditation system. The remaining core disciplines and the majority of subspecialties implemented the milestones starting in July 2014. We have now reached an important ‘‘milestone’’ in the implementation process, and our commentary proDOI: http://dx.doi.org/10.4300/JGME-07-03-43

‘We might as well be speaking different languages’: an innovative interprofessional education tool to teach and assess communication skills critical to patient safety

IntroductionAll practicing health professionals must be able to communicate effectively with their colleagues around the care of patients. Better communication between health professionals not only improves patient outcomes, but also...

Online assessment in medical education-current trends and future directions.

Online assessment in medical education-current trends and future directions.

医学教育における学習者の評価(1) 総論

医学教育における学習者の評価(1) 総論

On the Appropriateness of Norm- and Criterion-referenced Assessments in Medical Education

On the Appropriateness of Norm- and Criterion-referenced Assessments in Medical Education