Assessment In Medical Education Research Articles

Readability as a Source of Measurement Error in Medical Education Assessment

Readability as a Source of Measurement Error in Medical Education Assessment

Thoughts about assessment in undergraduate Medical Education

No abstract available.

Necessary but not sufficient: identifying conditions for effective feedback during internal medicine residents' clinical education.

Competency-based medical education and programmatic assessment intend to increase the opportunities for meaningful feedback, yet these conversations remain elusive. By comparing resident and faculty perceptions of feedback opportunities within one internal medicine residency training program, we sought to understand whether and how principles underlying meaningful feedback could be supported or constrained across a variety of feedback opportunities. Using case-study qualitative methodology, interviews and focus groups were conducted to explore 19 internal medicine residents' and 7 faculty members' perceptions of feedback across a variety of feedback opportunities: coaching, mini-CEXs, in-training evaluation reports and routine clinical supervision. Our data analysis moved iteratively between developing conceptual understandings and fine-grained analyses, while attending to both deductive and inductive analysis. Our results suggest that all feedback opportunities, including those created through formalized assessments, can foster meaningful feedback if faculty establish a trusting relationship with the resident, base their feedback on direct observation and support resident learning. However, formalized assessments were often perceived as inhibiting the conditions for meaningful feedback. A coaching program provided a context in which meaningful feedback could arise, in part because faculty were supported in shifting their focus from patient to resident. Meaningful feedback in clinical education may be fostered across a variety of feedback opportunities, however, it is often constrained by assessment. We must consider whether increasing the frequency of formative assessments may inhibit efforts to improve our feedback cultures while, in contrast, freeing up faculty to focus on supporting resident learning could improve these cultures.

What attributes define excellence in a trauma team? A qualitative study

Hospital trauma teams consist of a diverse spectrum of health care professionals who work together to deliver quality care. Although the qualities of a well-performing trauma team are often believed to be self-evident, there is little objective information about the most desirable personal and professional characteristics associated with quality trauma care. The aim of this study was to determine the traits and characteristics deemed of greatest value for a trauma team leader and a trauma team member in the adult trauma care setting. Semistructured interviews were conducted with trauma team leaders and trauma team members at a tertiary Canadian trauma centre. Standard qualitative research methodology was used. Interviews were recorded, transcribed and analyzed via an inductive analysis approach. Thematic saturation was achieved after 5 interviews, and 6 further interviews were conducted to ensure that a breadth of trauma care disciplines were included. Six attributes were identified to be of greatest value for trauma team leaders: communication, role clarity, experience, anticipation, management and decisiveness. Four attributes were identified to be of greatest value for trauma team members: engagement, efficiency, experience and collaboration. We further characterized the language defining the ranking of performance for each of these attributes. Results of this qualitative study involving an experienced and diverse spectrum of trauma team practitioners provide insight into the characteristics that are critical to establishing a “good” trauma team. These findings can be used to inform future determinations of the quality of trauma teams, the education of trauma practitioners and continuing medical education training and assessment tools.

Optimizing assessors' mental workload in rater-based assessment: acritical narrative review.

IntroductionRater-based assessment has resulted in high cognitive demands for assessors within the education of health professionals. Rating quality may be influenced by the mental workload required of assessors to complete rating tasks. The objective of this review was to explore interventions or strategies aimed at measuring and reducing mental workload for improvement in assessment outcomes in health professions education.MethodsA critical narrative review was conducted for English-language articles using the databases PubMed, EMBASE, and Google Scholar from conception until November 2018. To be included, articles were eligible if they reported results of interventions aimed at measuring or reducing mental workload in rater-based assessment.ResultsA total of six articles were included in the review. All studies were conducted in simulation settings (OSCEs or videotaped interactions). Of the four studies that measured mental workload, none found any reduction in mental workload as demonstrated by objective secondary task performance after interventions of assessor training or reductions in competency dimension assessment. Reductions in competency dimensions, however, did result in improvements in assessment quality across three studies.DiscussionThe concept of mental workload in assessment in medical education needs further exploration, including investigation into valid measures of assessors’ mental workload. It appears that adjusting raters’ focus may be a valid strategy to improve assessment outcomes. Future research should be designed to inform how to best reduce load in assessments to improve quality, while balancing the type and quantity of data needed for judgments.

Technology Enabled Formative Assessment in Medical Education: A Pilot Study through Kahoot

Technology Enabled Formative Assessment in Medical Education: A Pilot Study through Kahoot

Three Goals of Assessment in Medical Education

Assessment is closely connected to learning since assessment allows a learner’s current competencies to be measured. The results of such a measurement can then be used to produce learning. There are three goals for assessment in medical education: assessment of learning, assessment for learning, and assessment as learning. These three goals serve different purposes and therefore differ in how they are carried out. They also require different approaches for the assessors. It is crucial that all three goals be balanced and attained lest the testing culture wax and the learning culture wane, resulting in a situation in which learners place too much emphasis on passing the tests and not enough emphasis on learning and growing. No single method of assessment can sufficiently achieve all three goals. One comprehensive approach to achieve as well as balance all three goals is to utilize programmatic assessment, in which various methods of assessment are employed based on their strengths and how they can cover each other’s limitations.

ANALYSIS OF THE PERFORMANCE OF MCQs AS A PART OF FORMATIVE ASSESSMENT FOR IST MBBS STUDENTS IN BIOCHEMISTRY

Introduction: Multiple Choice Questions (MCQs) is one of the most preferred tool of assessment in medical education as a part of formative as well as summative assessment. MCQ performance as an assessment tool can be statistically analysed by Item analysis. Thus, aim of this study is to assess the quality of MCQs by item analysis and identify the valid test items to be included in the question bank for further use.
 Materials and methods: Formative assessment of Ist MBBS students was carried out with 40 MCQs as a part of internal examination in Biochemistry. Item analysis was done by calculating Difficulty index (P), Discrimination index (d) and number of Non- functional distractors.
 Results: Difficulty index (P) of 65% (26) items was well within acceptable range, 7.5% (3) items were too difficult whereas 27.5% (11) items were in the category of too easy. Discrimination Index (d) of 70% (28) items fell in recommended category whereas 10% (4) items were with acceptable, and 20% (8) were with poor Discrimination index. Out of 120 distractors 88.33% (106) were functional distractors and 11.66% (14) were non-functional distractors. After considering difficulty index, discrimination index and distractor effectiveness, 42.5% (17) items were found ideal to be included in the question bank.
 Conclusion: Item analysis remains an essential tool to be practiced regularly to improve the quality of the assessment methods as well as a tool for obtaining feedback for the instructors.
 Key Words: Difficulty index, Discrimination index, Item analysis, Multiple choice questions, Non-functional distractors

Can Teachers Distinguish Competencies From Entrustable Professional Activities?

There has been a recent transition from the use of "competencies" to "entrustable professional activities" (EPAs) in medical education assessment paradigms. Although this transition proceeds apace, few studies have examined these concepts in a practical context. Our study sought to examine how distinct the concepts of competencies and EPAs were to front-line clinical educators. A 20-item survey tool was developed based on the University of Calgary Department of Family Medicine's publicly available lists of competencies and EPAs. This tool required participants to identify given items as either a competency or an EPA, after reading a description of each. The tool was administered to a convenience sample of consenting clinical educators at 5 of the 14 training sites at the University of Toronto Department of Family and Community Medicine in 2018. We also collected information on years in practice, hours spent supervising per week, and direct involvement in medical education. We analyzed a total of 60 surveys. The mean rate of correct responses was 45.3% (+/- 21.8%). Subgroup analysis failed to reveal any correlation between any of the secondary characteristics and correct responses. Clinical educators in our study were not able to distinguish between competencies and EPAs. Further research is recommended prior to intensive curricular changes.

When I say … programmatic assessment in postgraduate medical education.

Turning a critical lens on the definition of programmatic assessment, Pearce and Prideaux argue that implementation in postgraduate training need not be as difficult as it seems if programs use programmatic thinking to connect elements already in place.

A prospective study on the efficacy of patient simulation in heart and lung auscultation

BackgroundThe use of simulation technology for skill training and assessment in medical education has progressively increased over the last decade. Nevertheless, the teaching efficacy of most technologies remains to be fully determined. The aim of this prospective study was to evaluate if a short individual training on a patient simulator could improve heart and lung auscultation skills in undergraduate students.MethodsA group of fifth-year medical school students, who had trained on a patient simulator in their third year (EXP, n = 55), was compared to a group of fifth-year medical school students who had not previously trained on it (CNT, n = 49). Students were recruited on a voluntary basis. Students were evaluated in terms of their ability to correctly identify three heart (II sound wide split, mitral regurgitation, aortic stenosis) and five lung sounds (coarse crackles, fine crackles, pleural rubs, rhonchi, wheezes), which were reproduced in a random order on the Kyoto-Kagaku patient simulator.ResultsExposure to patient simulator significantly improved heart auscultation skills, as mitral regurgitation was correctly recognized by 89.7% of EXP students as compared to 71.4% of CNT students (p = 0.02). In addition, a significantly greater percentage of EXP students correctly graphed all the heart diagnoses as compared to CNT students. There were no differences between the groups in lung auscultation.ConclusionsThis study demonstrates that training medical students with a patient simulator, individually for one hour, significantly ameliorated their heart auscultation skills. Our data suggests that patient simulation might be useful for learning auscultation skills, especially when it is combined with graphic sound display.

Comprehensive analysis of theory exam question papers of 2nd MBBS Pharmacology subject in Rajiv Gandhi University of Health Sciences, Karnataka, India

Background: Assessment is the backbone of curriculum development and an important component of medical education. Written exams are frequently used in medical education and question papers are important instruments of assessment of medical education. The present study is summative question paper based analysis study where the content validity, level of cognition, cognitive domains, marks distribution and time weightage in theory question papers were analyzed.Methods: Retrospective analysis of II MBBS pharmacology question papers (n=18) of RGUHS, Bengaluru from the year 2010 to 2018 was done. The data was analyzed according to modified Blooms levels of taxonomy, cognitive domains, areas of importance and weightage given to different topics in Paper 1 and 2 in comparison to university syllabus and with regard to time allotted in university syllabus.Results: Analysis of question papers revealed that topics like GIT, respiratory system, uterine drugs, antiseptics and disinfectants, immunosuppressant were consistently underrepresented in question papers. Also, majority of questions (68.06%) did not have any verbs associated with Blooms level of taxonomy and hence did not meet criteria of standard question paper. Comparison of time allocation to weightage of topics in syllabus revealed no significant variations in percentage distribution.Conclusions: Assessment in medical education should not be used merely for classification, grading and certification rather it should become an instrument for promoting growth of knowledge. For this the university should frame standard blueprint for question paper setters and the question paper setters should follow those guidelines strictly and ensure that weightage given for different topics is maintained.

A Study to Evaluate the Effectiveness of Open Book Tests Over Close Book Tests and Perception of Medical Students in Physiology


 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Introduction
 At present, the Medical knowledge is expanding and changing rapidly and there is increasing need to concentrate on education based on competencies. Closed book exam is an established approach to assessment in medical education which is widely accepted method of superficial learning. Adding open book test to the formative assessment can assist in handling this growth of knowledge.
 Objectives
 To evaluate the effectiveness of open book test on student's performance in summative assessment and to know the perception of students on open book test.
 Methodology
 The study was conducted on I MBBS students (n=150), who were divided into 2 groups, Group I: subjected to open book test for 4 weeks and Group II: subjected to closed book tests for 4weeks. There was a cross-over of the two groups and Group I were subjected to the closed book test and Group II students were subjected to open book tests. The mean marks obtained in summative assessment were statistically compared. Perception of the students was taken through feedback questionnaire.
 Results
 The mean marks obtained by the students (n= 141) in the summative assessment after open book test (Mean± SD =26.37±6.235) are statistically significant than those after closed book tests(mean± SD =24.09± 6.3513(p value< 0.005).
 Conclusion
 71% of students recommended open book tests for the formative assessment. Hence, adding open book test to the medical curriculum along with regularly used closed book test are expected to stimulate deep learning.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

The Relationship between Performance of Students in Internal Assessment and Final Summative Assessment in the Subject of Physiology in a North Indian Medical College


 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Background
 Assessments, if suitably conducted, can serve several goals and provide many advantages to all stakeholders-the medical students, the teaching faculty, the universities and the society as a whole. Assessment in medical education is mostly of two forms either formative or summative. Although it is generally assumed that the performance of a student in formative assessment will act as a predictor for performance in the summative examination not many studies are available support it.
 Objectives
 To explore whether performance in the continuous internal assessment, correlates with the final summative assessment in first year medical students in the subject of Physiology for the last three years.
 Materials and Method
 This study was conducted over a period of six months at SKIMS medical college, Srinagar. It was a observational record-based study on 280 students who took a thorough internal assessment and final examination in the subject of physiology in the batches 2014–15, 2015–16 and 2016–17.
 Results
 The internal assessment marks showed a positive correlation with marks obtained in final assessment or final university exams in all three batches, which is statistically highly significant (p<0.01). It was also observed that the percentage of final university marks was higher as compared to internal assessment marks in all batches.
 Conclusion
 The present study revealed a positive correlation between internal assessment and final university marks supporting the general assumption that the performance of a student in formative assessment will act as a predictor for performance in the final summative examination.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Using Kahoot! as a formative assessment tool in medical education: a phenomenological study

BackgroundGamification is an increasingly common phenomenon in education. It is a technique to facilitate formative assessment and to promote student learning. It has been shown to be more effective than traditional methods. This phenomenological study was conducted to explore the advantages of gamification through the use of the Kahoot! platform for formative assessment in medical education.MethodsThis study employed a phenomenological design. Five focus groups were conducted with medical students who had participated in several Kahoot! sessions.ResultsThirty-six categories and nine sub-themes emerged from the focus group discussions. They were grouped into three themes: attractive learning tool, learning guidance and source of motivation.ConclusionsThe results suggest that Kahoot! sessions motivate students to study, to determine the subject matter that needs to be studied and to be aware of what they have learned. Thus, the platform is a promising tool for formative assessment in medical education.

A New Competency-Based Instrument to Assess Resident Knowledge and Self-Efficacy in Primary Palliative Care.

There is a need to improve both primary palliative care (PPC) education and its assessment in graduate medical education (GME). We developed an instrument based on published palliative care (PC) competencies to assess resident competency and educational interventions. To describe the development and psychometric properties of a novel, competency-based instrument to measure resident knowledge and self-efficacy in PPC. We created a 2-part instrument comprised of a knowledge test (KT) and a self-efficacy inventory (SEI) addressing 18 consensus, core PC resident competencies across 5 domains: pain and symptom management; communication; psychosocial, spiritual, and cultural aspects of care; terminal care and bereavement; and PC principles and practice. The instrument was distributed to 341 internal medicine residents during academic years 2015 to 2016 and 2016 to 2017. A standard item analysis was performed on the KT. Internal consistency (Cronbach α) and variable relationships (factor analysis) were measured for the SEI. One hundred forty-four residents completed the survey (42% response). For 15 KT items, difficulty ranged from 0.17 to 0.98, with 7 items ranging 0.20 to 0.80 (typical optimum difficulty); discrimination ranged from 0.03 to 0.60 with 10 items ≥0.27 (good to very good discrimination). Cronbach α was 0.954 for 35 SEI items. Factor analysis of combined 2015 to 2016 items yielded 4 factors explaining the majority of variance for the entire set of variables. Our instrument demonstrates promising psychometric properties and reliability in probing the constructs of PC and can be further utilized in PC GME research to assess learners and evaluate PPC educational interventions.

A Competency-Guided Veterinary Curriculum Review Process

Competencies can guide outcomes assessment in veterinary medical education by providing a core set of specific abilities expected of new veterinary graduates. A competency-guided evaluation of Colorado State University's (CSU) equine veterinary curriculum was undertaken via an alumni survey. Published competencies for equine veterinary graduates were used to develop the survey, which was distributed to large animal alumni from CSU's Doctor of Veterinary Medicine program. The results of the survey indicated areas for improvement, specifically in equine business, surgery, dentistry, and radiology. The desire for more hands-on experiences in their training was repeatedly mentioned by alumni, with the largest discrepancies between didactic knowledge and hands-on skills in the areas of business and equine surgery. Alumni surveys allow graduates to voice their perceived levels of preparation by the veterinary program and should be used to inform curriculum revisions. It is proposed that the definition and utilization of competencies in each phase of a curricular review process (outcomes assessment, curriculum mapping, and curricular modifications), in addition to faculty experience and internal review, is warranted.

Using Mobile Apps for Health Management: A New Health Care Mode in China

BackgroundChina has a large population; however, medical resources are unevenly distributed and extremely limited, and more medical services are needed. With the development and ever-increasing popularity of mobile internet communication, China has created a mode of mobile health (mHealth) care to resolve this problem.ObjectiveThe aim of this study was (1) to describe the problems associated with China’s medical care practice, (2) explore the need for and the feasibility of internet-based medical care in China, and (3) analyze the functionality of and services offered by internet-based health care platforms for the management of chronic diseases.MethodsData search was performed by searching national websites, the popular search engine Baidu, the App Store, and websites of internet medical care institutions, using search terms like “mobile health,” “Internet health,” “mobile medical,” “Internet medical,” “digital medical,” “digital health,” and “online doctor.” A total of 6 mobile apps and websites with the biggest enrollment targeting doctors and end users with chronic diseases in China were selected.ResultsWe recognized the limitations of medical and health care providers and unequal distribution of medical resources in China. An mHealth care platform is a novel and efficient way for doctors and patients to follow up and manage chronic diseases. Services offered by these platforms include reservation and payment, medical consultation, medical education assessment, pharmaceutical and medical instruments sales, electronic medical records, and chronic disease management. China’s health policies are now strongly promoting the implementation of mHealth solutions, particularly in response to the increasing burden of chronic diseases and aging in the population.ConclusionsChina's internet-based medical and health care mode can benefit the populace by providing people with high-quality medical resources. This can help other countries and regions with high population density and unevenly distributed medical resources manage their health care concerns.

Survey of medical student knowledge in oncology and survivorship care in Nigeria: A pilot study.

e18106 Background: In low middle-income countries (LMICs), cancer patients present in later stages of disease for many reasons, including missed diagnoses. Due to high patient: physician ratios, generalists are taught broadly and prepared to treat diseases outside the scope of their interest in training. The delivery of high-quality cancer care requires an interdisciplinary team of physicians with excellent medical school training in oncology. To date, there has been no formal assessment of medical education for oncology in Nigeria. This study aims to assess the ability of the curriculum in Nigeria to prepare future doctors tasked with treating a growing number of patients with cancer. Methods: An anonymous survey was administered to final year medical students at Lagos State University and University of Ibadan, Nigeria. The survey assessed demographics, reports of experience during medical school with oncology, comfort ratings with different aspects of oncologic care (Likert scale, 1 = not at all, 5 = extremely), and included five clinical vignettes. Results: 146 graduating students in two medical schools responded (response rate = 36.6%). Twelve percent of students indicated that they planned to pursue a career in oncology. The majority of students reported exposure through lectures to medical oncology (85%), radiation oncology (82%), and surgical oncology (85%), but fewer reported exposure to lectures in hospice care (75%) and cancer survivorship care (52%). Most students (87%) have not attended a tumor board during medical school. In clinical vignettes, students performed worst on long-term chemotherapy toxicities, and best on surgical oncology risk. Overall, students indicated being “moderately comfortable” with topics in medical, radiation, surgical, and palliative care on a Likert scale, and “somewhat comfortable” with survivorship care. Conclusions: Although students in Nigeria report learning about disciplines in oncology care, they report varying comfort levels with specific oncology topics by the end of medical school. The results of this pilot study support the need to develop an interdisciplinary oncology curriculum in Nigeria.

Use of work-based assessments in postgraduate medical education.

Work-based assessments are ubiquitous in postgraduate medical training in the UK. This article discusses the variety of these assessments and explores barriers to their use and solutions for improving the educational value of these tools for adult learners. The focus should be on feedback and learning rather than assessment, and this may promote discussion of more challenging scenarios where the opportunity for learning is greater. Mobile devices may help reduce the administrative, geographical and time constraints of completing work-based assessments.