Multiple-choice Research Articles

Consensus and controversies on post-acute care decision making and referral to geriatric rehabilitation: A national survey

BackgroundTransitioning older hospital patients to the appropriate type of post-acute care has become an urgent clinical issue within the context of changing demographics and limited duration of hospital stay. ObjectiveConsensus on assessments that guide post-acute care decision making would benefit potential patients and support cooperation between settings. DesignA national web-based questionnaire focusing on professional contributions, patient involvement and the use of triage items and measures. ParticipantsHospital and geriatric rehabilitation professionals in the Netherlands participated as respondent groups, representing ‘sending’ and ‘receiving’ professionals. MethodsA comprehensive questionnaire was used with open, multiple choice and closed questions, exploring in detail how assessment of hospital patients in need of a post-acute care decision was performed. Descriptive statistics were applied together with deductive coding of qualitative data. ResultsA total of 104 hospital liaison nurses (66.7 %) and 52 GR professionals (33.3 %) participated. Respondents were reasonably satisfied with the current triage practice. Hospital liaison nurses valued their operational responsibility for triage. Geriatric rehabilitation professionals wanted active involvement in decision making and deemed hospital paramedic expertise sub-optimally applied. ‘Too little involvement’ of patients and families was felt by 50.0 % of the GR respondents versus 15.5 % of hospital respondents. The importance of half (47.8 %) of the triage items was rated differently between respondent groups. When discussing complex cases between sending and receiving professionals, views were felt as complementary. ConclusionsBoth sending and receiving professionals expressed moderate satisfaction with post-acute care decision making, whereas their views on triage assessments differed according to setting and role. The patients’ voice may be insufficiently heard in triage decisions. Shared expertise and a consensual approach can develop when triage consultation is facilitated by both hospitals and PAC facilities. This study offers ingredients to reach a multi-professional view on post-acute care decision making and referral to geriatric rehabilitation.

Challenges of patients with intrathecal baclofen pump therapy during COVID-19 pandemic in Saudi Arabia

Background: This study was carried out to analyze the effects of the COVID-19 pandemic on patients with intrathecal baclofen (ITB) therapy. Methods: Patients who came for ITB refill during the COVID-19 pandemic between March 2020 and August 2020 were contacted, and a survey was conducted regarding challenges faced during the COVID-19 pandemic. Data obtained from the chart review included demographics, diagnoses, pump information, baclofen dose, and COVID triage. Information was collected regarding travel arrangements, appointments, and problems related to ITB therapy or travel. The survey included 12 multiple-choice questions related to travel, appointment scheduling, and difficulties faced during the travel restrictions. Results: The study included 19 patients with ITB (mean age: 38.1±14.4) from outside Riyadh city who came for ITB refill during the COVID-19 pandemic. The majority of the patients were males. Spinal cord injury with paraplegia was the most common diagnosis (52.6%). In total, 7 patients had either clinical (31.6%) or travel-related difficulties (15.8%). No patient had ITB withdrawal at the time of the clinic visit. Nearly half (47.4%) of the patients had to change their appointment for refill. The percentage of patients who used a car as a mode of travel pre-COVID was 42.1%, which increased to 78.9% during the pandemic. They traveled 747.4±356.6 km (7.4±3.7 h) to reach the hospital for refill. Patients needed 1–4 accompanied persons for assistance during travel, with the majority (68.4%) requiring 2 accompanied persons. Statistical analysis did not reveal any significant associations between clinical or travel difficulties and appointment changes. Conclusions: A pandemic or similar disaster can pose challenges to patients on ITB therapy living in areas with limited access to specialized care. In this study, none of the patients had serious complications or adverse outcomes; however, the main difficulties were attributed to travel. It is crucial to establish strategies at institutional, regional, or national levels to formulate guidelines for managing patients with ITB therapy during a pandemic.

Which Encoding Techniques Facilitate Comprehension?

Previous work suggests that similar cognitive processes contribute to memory and comprehension. This is unsurprising as both begin with a common process: encoding. Despite this, the investigation of techniques that benefit memory and comprehension has proceeded separately. In the current study, we compared the robust memory techniques of production and drawing to a similarly effective comprehension strategy known as paraphrasing. Depending on the group, participants were asked to either engage in one of the encoding types (read aloud, draw, or paraphrase) or to silently read 20 term-definition pairs (randomly intermixed and counterbalanced). The encoding techniques of drawing and paraphrasing resulted in better performance on a multiple-choice test of concept comprehension, relative to silently reading. By contrast, reading aloud at encoding did not lead to any benefit relative to silently reading. The results suggest that techniques that invoke transformation of the to-be-remembered text into another format, be it into a picture (drawing) or personally relevant summary (paraphrasing), are particularly effective at improving comprehension. By contrast, encoding techniques that mainly provide a perceptual repetition (production and silent reading) are less effective.

ChatGPT-4’s capability in addressing multiple-choice questions within the primary examination of the Australian and New Zealand College of Anaesthetists

ChatGPT-4’s capability in addressing multiple-choice questions within the primary examination of the Australian and New Zealand College of Anaesthetists

Comparative Assessment of Otolaryngology Knowledge Among Large Language Models

ObjectiveThe purpose of this study was to evaluate the performance of advanced large language models from OpenAI (GPT‐3.5 and GPT‐4), Google (PaLM2 and MedPaLM), and an open source model from Meta (Llama3:70b) in answering clinical test multiple choice questions in the field of otolaryngology—head and neck surgery.MethodsA dataset of 4566 otolaryngology questions was used; each model was provided a standardized prompt followed by a question. One hundred questions that were answered incorrectly by all models were further interrogated to gain insight into the causes of incorrect answers.ResultsGPT4 was the most accurate, correctly answering 3520 of 4566 questions (77.1%). MedPaLM correctly answered 3223 of 4566 (70.6%) questions, while llama3:70b, GPT3.5, and PaLM2 were correct on 3052 of 4566 (66.8%), 2672 of 4566 (58.5%), and 2583 of 4566 (56.5%) questions. Three hundred and sixty‐nine questions were answered incorrectly by all models. Prompts to provide reasoning improved accuracy in all models: GPT4 changed from incorrect to correct answer 31% of the time, while GPT3.5, Llama3, PaLM2, and MedPaLM corrected their responses 25%, 18%, 19%, and 17% of the time, respectively.ConclusionLarge language models vary in their understanding of otolaryngology‐specific clinical knowledge. OpenAI's GPT4 has a strong understanding of core concepts as well as detailed information in the field of otolaryngology. Its baseline understanding in this field makes it well‐suited to serve in roles related to head and neck surgery education provided that the appropriate precautions are taken and potential limitations are understood.Level of EvidenceN/A Laryngoscope, 2024

Pain Education in Postgraduate Medical Curriculam in Bangladesh: An In-depth Analysis

Background: Pain is one of the most common symptoms for which people seek help from physicians. But it is not well emphasized over curriculum and evaluation method of medical, dental or nursing curriculum. The objective of the present study was to evaluate pain education in postgraduate medical, dental and nursing curricula in Bangladesh.Methods: This is a descriptive and analytical study which evaluated post-graduate medical education. For building the matrix, a graphical representation “concept map” was formed. Key words from this map were used for evaluation.Results: : In postgraduate curriculum, from 2014 to 2018, highest 48% marks were allocated in residency program of Physical medicine. Short answer questions and multiple choice questions were used for assessment of knowledge part of pain education. To evaluate skill of pain management objective structured clinical examination was used.Conclusion: Dedicated pain curricula with purpose, planning for implementation, encouraging to learners can make a difference to motivate healthcare professional for better pain management situation in Bangladesh.

Capabilities of GPT-4 in ophthalmology: an analysis of model entropy and progress towards human-level medical question answering

BackgroundEvidence on the performance of Generative Pre-trained Transformer 4 (GPT-4), a large language model (LLM), in the ophthalmology question-answering domain is needed.MethodsWe tested GPT-4 on two 260-question multiple choice question...

A Strategic Approach to Succeed on Clinical Case-Based Multiple-Choice Exams

Despite the importance of case-based multiple-choice question (CBMCQ) exams, medical educators rarely discuss strategies to systematically approach these questions, and literature on the topic is limited. Through trial-and-error, many students discover or develop more refined and efficient approaches to answering CBMCQs that help them maximize the application of their knowledge base. In this article we present a structured approach to answering CBMCQs, including meta-cognitive strategies to understand the reasons for errors. We provide strategies for success on CBMCQ-based exams and approaches to challenging question types. We also present tips for helping neurodiverse students and an approach for medical educators to teach these skills to their learners.

Evaluating Performance of ChatGPT on MKSAP Cardiology Board Review Questions

Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing tool created by OpenAI. Much of the discussion regarding artificial intelligence (AI) in medicine is the ability of the language to enhance medical practice, improve efficiency and decrease errors. The objective of this study was to analyze the ability of ChatGPT to answer board-style cardiovascular medicine questions by using the Medical Knowledge Self-Assessment Program (MKSAP).The study evaluated the performance of ChatGPT (versions 3.5 and 4), alongside internal medicine residents and internal medicine and cardiology attendings, in answering 98 multiple-choice questions (MCQs) from the Cardiovascular Medicine Chapter of MKSAP. ChatGPT-4 demonstrated an accuracy of 74.5 %, comparable to internal medicine (IM) intern (63.3 %), senior resident (63.3 %), internal medicine attending physician (62.2 %), and ChatGPT-3.5 (64.3 %) but significantly lower than cardiology attending physician (85.7 %). Subcategory analysis revealed no statistical difference between ChatGPT and physicians, except in valvular heart disease where cardiology attending outperformed ChatGPT (p = 0.031) for version 3.5, and for heart failure (p = 0.046) where ChatGPT-4 outperformed senior resident. While ChatGPT shows promise in certain subcategories, in order to establish AI as a reliable educational tool for medical professionals, performance of ChatGPT will likely need to surpass the accuracy of instructors, ideally achieving the near-perfect score on posed questions.

Understanding physician preferences about combined thrombolysis and thrombectomy in patients with large vessel occlusion: An international cross-sectional survey

BackgroundA recently published individual participant-level meta-analysis found that EVT alone was not non-inferior to combined intravenous thrombolysis (IVT) and EVT. Our aim was to determine factors that influence physicians’ treatment choice of IVT-alone versus EVT-alone versus a combined approach. MethodsWe performed an international, structured, invite-only survey among physicians treating patients presenting with AIS. Respondents were asked 16 multiple choice questions. Fourteen questions involved the respondent being provided with a clinical scenario. In each scenario, a patient was presenting with an AIS with LVO, varying a single clinical or imaging feature. ResultsA total of 282 stroke physicians (mean age 46 years, 75 % males) participated in the survey. In LVO stroke, eligible for both IVT and EVT, without other qualifiers, 220 (85.9 %) respondents chose to pursue a combined approach. For age over 80 years, 191 (74 %) participants opted for combined approach, which decreased to 121 (48.2 %) with dementia and 148 (57.4 %) if the patient was on dual anti-platelet therapy (DAPT). Of respondents choosing combination therapy in a patient above the age of 80, only 105 (56.8 %) would pursue the same in a patient with dementia. For imaging factors, 177 (72.8 %) opted for a combined approach for intracranial carotid occlusion, which decreased to 160 (65.3 %) in tandem occlusions. Overall, 88 (38 %) respondents agreed to the statement “I am uncomfortable with uncertainty in patient care”. ConclusionsIn a typical patient with AIS due to LVO, most respondents still choose a combined revascularization approach but discrepancy in decision-making increases in complex scenarios.

EFFECTIVENESS OF SELECTED SONGS ON SPOTIFY TO CONTINUOUS TENSES MASTERY THROUGH DEDUCTIVE APPROACH

This study aims to examine the effect of selected songs on Spotify to continuous tense mastery through a deductive approach. Conducted as a quantitative research employing a pre-experimental design with one group pre-test post-test, it focused on ninth-grade students of SMP Negeri 25 Pontianak Utara in the Academic Year 2023/2024. Data collection involved a multiple-choice test comprising 12 questions, validated and deemed reliable through reliability and validity tests. Results indicated a significant enhancement in students' scores, with a pre-test mean score of 61.79 and a post-test mean score of 91.38. The t-test value of 8.508, surpassing the critical t-table value of 1.701, affirmed the alternative hypothesis. Moreover, the effect size of 1.58 signified strong effectiveness. Consequently, the study concluded that selected Spotify songs effectively facilitate continuous tense mastery through deductive methods. This research highlights opportunities for further exploration, suggesting investigation in diverse educational settings or with different student demographics to validate the efficacy of using selected songs for mastering other grammatical tenses.

Pengaruh Media Scrapbook Terhadap Kemampuan Membaca Pemahaman Siswa Kelas IV UPT SDN 060816 Medan

This research aims to determine the effect of scrapbook media on the reading comprehension ability of class IV students at UPT SDN 060816 Medan. This research has several problems, namely when students read they do not understand the content of the reading they read. This research is quantitative research with a quasi-experimental type of research. The sample in this study used a saturated sampling technique consisting of two classes, namely class IV A with 32 students as the experimental class using scrapbook media and class IV B with 31 students as the control class without using media. The data collection technique used in this research is a multiple choice test. The research results from both classes, namely the experimental class, had an average score of 45.81 (pretest) and 82.47 (posttest). And the control class has an average of 39.81 (pretest) and 67.90 (posttest). The results of the hypothesis test are the Paired Sample t-Test with a significance level of α = 0.05. The results found a sig (2-tailed) value of 0.000 < 0.05. So it can be concluded that Ho is rejected and Ha is accepted, so there is an influence of scrapbook media on the reading comprehension ability of class IV students at UPT SDN 060816 Medan.

Pengembangan Bahan Ajar Berbasis Pemecahan Masalah untuk Meningkatkan Kemampuan Berpikir Kritis Peserta Didik pada Mata Pelajaran PPKn Kelas IV

This research aims to develop problem-solving-based teaching materials to improve students' critical thinking skills in class IV PPKn subjects based on the fact that the use of teaching materials in each PPKn subject is rarely used because educators only rely on textbooks as a guide. And in improving students' critical thinking skills, teachers only use multiple choice tests in the textbook without asking students to create their own ideas from the knowledge they have obtained. This research uses the ADDIE model (Analysis, Design, Development, Implementation, Evaluation) but the focus of this research is limited to this stage Development just. Data collection techniques used were interviews, observation, questionnaires and documentation. The results obtained after the teaching materials were validated by 3 validator experts, namely material experts, obtained an average score of 89.3% with the "Very Eligible" criteria. The results of the media expert validation assessment obtained an average score of 94.3% with the criteria "Very Eligible". And the results of the linguist validation assessment obtained an average score of 82% with the criteria "Very Eligible". Meanwhile, the results of the teacher's assessment of problem-solving based teaching materials obtained an average score of 94.6% with the criteria "Very Appropriate". This shows that problem-solving based teaching materials are very suitable for use in the learning process to improve students' critical thinking abilities.

Comparison of Gemini Advanced and ChatGPT 4.0's Performances on the Ophthalmology Resident Ophthalmic Knowledge Assessment Program (OKAP) Examination Review Question Banks.

Background With advancements in natural language processing, tools such as Chat Generative Pre-Trained Transformers (ChatGPT) version 4.0 and Google Bard's Gemini Advanced are being increasingly evaluated for their potential in various medical applications. The objective of this study was to systematically assess the performance of these language learning models (LLMs) on both image and non-image-based questions within the specialized field of Ophthalmology. We used a review question bank for the Ophthalmic Knowledge Assessment Program (OKAP) used by ophthalmology residents nationally to prepare for the Ophthalmology Board Exam to assess the accuracy and performance of ChatGPT and Gemini Advanced. Methodology A total of 260 randomly generated multiple-choice questions from the OphthoQuestions question bank were run through ChatGPT and Gemini Advanced. A simulated 260-question OKAP examination was created at random from the bank. Question-specific data were analyzed, including overall percent correct, subspecialty accuracy, whether the question was "high yield," difficulty (1-4), and question type (e.g., image, text). To compare the performance of ChatGPT-4 and Gemini on the difficulty of questions, we utilized the standard deviation of user answer choices to determine question difficulty. In this study, a statistical analysis of Google Sheets was conducted using two-tailed t-tests with unequal variance to compare the performance of ChatGPT-4.0 and Google's Gemini Advanced across various question types, subspecialties, and difficulty levels. Results In total, 259 of the 260 questions were included in the study as one question used a video that any form of ChatGPT could not interpret as of May 1, 2024. For text-only questions, ChatGPT-4.0.0 correctly answered 57.14% (148/259, p < 0.018), and Gemini Advanced correctly answered 46.72% (121/259, p < 0.018). Both versions answered most questions without a prompt and would have received a below-average score on the OKAP. Moreover, there were 27 questions utilizing a secondary prompt in ChatGPT-4.0 compared to 67 questions in Gemini Advanced. ChatGPT-4.0 performed 68.99% on easier questions (<2 on a scale from 1-4) and 44.96% on harder questions (>2 on a scale from 1-4). On the other hand, Gemini Advanced performed 49.61% on easier questions (<2 on a scale from 1-4) and 44.19% on harder questions (>2 on a scale from 1-4). There was a statistically significant difference in accuracy between ChatGPT-4.0 and Gemini Advanced for easy (p < 0.0015) but not for hard (p < 0.55) questions. For image-only questions, ChatGPT-4.0 correctly answered 39.58% (19/48, p < 0.013), and Gemini Advanced correctly answered 33.33% (16/48, p < 0.022), resulting in a statistically insignificant difference in accuracy between ChatGPT-4.0 and Gemini Advanced (p < 0.530). A comparison against text-only and image-based questions demonstrated a statistically significant difference in accuracy for both ChatGPT-4.0 (p < 0.013) and Gemini Advanced (p < 0.022). Conclusions This study provides evidence that ChatGPT-4.0 performs better on the OKAP-style exams and is improved compared to Gemini Advanced within the context of ophthalmic multiple-choice questions. This may show an opportunity for increased worth for ChatGPT in ophthalmic medical education. While showing promise within medical education, caution should be used as a more detailed evaluation of reliability is needed.

'I have to remind myself that everyone's search is different': experiences and outcomes of searching and not searching for donor connections among donor conceived adults.

What are the experiences and outcomes of donor conceived adults who are actively searching for, open to contact with, or not searching for donor connections? Most participants were actively searching or open to contact, and 67% had found or been found by a connection; finding or not finding experiences were complex. There is variation among donor conceived individuals in their interest in donor connections. Individual reasons for searching for connections, and which donor connections are searched for, also vary. Most research studies have focussed on individuals who are actively searching for their donor or donor siblings. Global increases in direct-to-consumer DNA testing and social media participation mean that connections may be made to individuals unaware of their (or their relatives') involvement with donor conception. These social and technological changes have also increased the chances of donor conceived individuals being contacted without expecting or desiring contact. This study included 88 donor conceived adults, in the UK, who participated in an online multi-method survey between January and August 2022. The survey was designed in consultation with staff and volunteers from the UK's largest community networks for donor conception families (Donor Conception Network, DCN) and donor conceived people (Donor Conceived Register Registrants' Panel, DCRRP). It was piloted by five donor conceived people before its launch. Participants were recruited with assistance from DCN and DCRRP, via social media, university mailing lists, and snowballing. Participants were mostly female (n = 65, 74%) and sperm donor conceived (n = 79, 90%). Of the 88 participants, 39 (44%) were actively searching for their donor connections, 44 (50%) were open to contact but not actively searching, and 5 (6%) were not searching. Questions were closed (yes/no, rating scale, or multiple choice) or open-ended, addressing experiences of donor conception, searching for connections, and finding or not finding connections. Data were analysed both quantitatively and qualitatively. Quantitative results showed no differences between the groups on any demographic variables or in when or how they found out about being donor conceived, and no differences between active searchers and those open to contact in whether they had found their donor connections. Significant differences were found between groups in their interest in their genetic history and the perceived importance of genetics to their sense of identity, with active searchers being more interested and rating this as more important than those open to contact. Methods of searching significantly differed across groups, with active searchers using genetic testing and social media more than those open to contact. 59 participants across all groups (active searchers (n = 29, 74%), open to contact (n = 27, 61%), not open to contact (n = 3, 60%)) had found or been found by a donor connection. Experiences of finding or not finding donor connections among participants actively searching or open to contact were captured by the theme complexities, with six subthemes: uncertainties in searching and relating; searching as open-ended; different donor connections, different experiences; expectations and realities; searching and finding or not finding as catalysing change; and experiences of other donor conceived people. Most participants were members of relevant community organizations. As is common in research in this area, the sample was mostly female and conceived using donor sperm. Donor conceived people who are disinterested in donor connections may be unlikely to participate in research on this topic. The nature and impact of the search process itself should be considered when developing appropriate mechanisms of support for all donor conceived people, regardless of whether they are actively searching for connections or not. Further research should seek to better understand how donor conceived people with varying levels of interest in searching for donor connections differ from one another. This study was funded by the UK Economic and Social Research Council [New Investigator Award ES/S015426/1]. The authors have no competing interests to declare. N/A.

New Metrics for Analysing Multiple-choice Questions

While the ideal of constructive alignment in curriculum is well established, and the importance of evaluating learning and teaching is well known, evaluating assessments remains a complex task and gaps can arise between learning outcomes, learning activities and assessments. Our study outlines an innovative way of analysing multiple-choice question (MCQ) examinations, which reveals possible weaknesses in the examination design and gaps in the alignment of curriculum. Individual examination questions are analysed through traditional metrics of the Discrimination Index (DI) and Difficulty Index (DIFF), combined with two novel metrics called the Grade Inversion Score (GIS) and Association with Total Score (ATS). Focusing on examination marks from a data science examination from The University of Sydney, we perform two investigations. First, we identify poorly designed questions using DI, DIFF, GIS and ATS. Second, as multiple questions arise as deficient in these metrics, we explore specific areas in the curriculum where there may be misalignment of course material with the learning outcomes. Our analysis provides a simple visual way for examiners to inspect the validity of an examination design and encourages an evidence-based approach to exploring curriculum alignment using multiple-choice assessments.

Working with partial hand prostheses: An investigation into experiences of clinicians

Background: Individuals living with partial hand absence often face challenges in performing everyday tasks and fully participating in society. Prosthetic devices offer a range of benefits, including improved functionality or cosmesis. However, partial hand devices can be rejected by users. In addition, there is a lack of evidence that establishes methods or factors influencing the clinical prescription of prosthetics specifically designed for this population. Objectives: The objectives of the investigation were to gain understanding of who is prescribing partial hand prosthetics and the factors which influence device selection. Study Design: A 36-item anonymous online survey was designed on Qualtrics and distributed to clinicians. The survey contained multiple-choice, Likert-type, closed, and open-ended questions. Methods: Quantitative and qualitative analyses were performed on Qualtrics and Microsoft Excel. Results: Thirty-seven clinicians from various occupations participated. The majority agreed that outcome measures are useful for assessing partial hand prosthetics but suggested room for improvement. Although clinicians use various outcome measurement tools, there is a lack of tools specifically tailored to this population. Factors such as existing function, occupation, and hobbies were identified as important when selecting prosthetic devices for users. Funding influenced the range of partial hand devices available for prescription. Conclusions: The prescription process involves multiple stakeholders. Various factors, including funding and user satisfaction, must be considered in the decision-making process. There is a lack of specific outcome measures recommended for evaluating prosthetics in this population. However, providing loaner units before final prescription yields positive outcomes.

The Effectiveness of Experiential Learning Strategy in Achieving Science Subject Competence Among Fifth Grade Elementary School Students

This study investigates the effectiveness of experiential learning strategies in enhancing science subject competence among fifth-grade elementary students in Anbar Governorate, where traditional teaching methods dominate. Prior research indicates a lack of engagement and critical thinking among students, emphasizing the need for pedagogical approaches that promote active learning and real-world experiences. Employing a descriptive and experimental design, this research included two groups: an experimental group receiving instruction through experiential learning and a control group taught via traditional methods. The sample consisted of 65 fifth-grade students from Al-Mu'min Primary School, chosen to ensure representativeness. Pre-experimental assessments ensured equivalence in variables such as age, intelligence, and previous academic achievement. The study's impact was evaluated using a post-test composed of 40 multiple-choice questions aligned with the science curriculum. Statistical analysis revealed a significant difference in mean achievement test scores between the groups, with the experimental group achieving a higher average score of 29.90 compared to the control group's 22.55. The calculated t-value was 10.11, surpassing the critical t-value of 2.00 at the 0.05 significance level, indicating strong evidence for the effectiveness of experiential learning strategies. Furthermore, the effect size measured using eta squared (η² = 0.63) confirmed a large impact on student achievement. The findings signify that experiential learning strategies substantially enhance science subject competence among elementary students, fostering greater engagement and understanding of scientific concepts. These results advocate for the integration of experiential learning methodologies in science education, reflecting a shift away from traditional teaching practices to more effective, student-centered approaches that promote critical thinking and real-world application of knowledge.

Experts’ prediction of item difficulty of multiple-choice questions in the Ethiopian Undergraduate Medicine Licensure Examination

BackgroundThe ability of an expert’s item difficulty ratings to predict test-taker actual performance is an important aspect of licensure examinations. Expert judgment is used as a primary source of information for users to make prior decisions to determine the pass rate of test takers. The nature of raters involved in predicting item difficulty is central to set credible standards. Therefore, this study aimed to assess and compare raters’ prediction and actual Multiple-Choice Questions’ difficulty of the undergraduate medicine licensure examination (UGMLE) in Ethiopia.Method815 examinees’ responses to 200 Multiple-Choice Questions (MCQs) were used in this study. The study also included experts’ item difficulty ratings of seven physicians who participated in the standard settings of UGMLE. Then, analysis was conducted to understand experts’ rating variation in predicting the actual difficulty levels of examinees. Descriptive statistics was used to profile the mean rater’s and actual difficulty value for MCQs, and ANOVA was used to compare the mean differences between raters’ prediction of item difficulty. Additionally, regression analysis was used to understand the interrater variations in item difficulty predictions compared to the actual difficulty. The proportion of variance of actual difficulty explained from rater prediction was computed using regression analysis.ResultsIn this study, the mean difference between raters’ prediction and examinees’ actual performance was inconsistent across the exam domains. The study revealed a statistically significant strong positive correlation between the actual and predicted item difficulty in exam domains eight and eleven. However, a non-statistically significant very weak positive correlation was reported in exam domains seven and twelve. The multiple comparison analysis showed significant differences in mean item difficulty ratings between raters. In the regression analysis, experts’ item difficulty ratings of the UGMLE had 33% power in predicting the actual difficulty level. The regression model also showed a moderate positive correlation (R = 0.57) that was statistically significant at F (6, 193) = 15.58, P = 0.001.ConclusionThis study demonstrated the complex process for assessing the difficulty level of MCQs in the UGMLE and emphasized the benefits of using experts’ ratings in advance. To ensure the exams maintain the necessary reliable and valid scores, raters’ accuracy on the UGMLE must be improved. To achieve this, techniques that align with the evolving assessment methodologies must be developed.

Two-Phase Individual Assessments: A Second-Chance Assessment Strategy With Individualized Feedback to Promote Assessment for Learning.

High-stakes multiple-choice question (MCQ) exams in medical education typically focus on assessment of learning at a single point without providing feedback for improvement. Educators can achieve a more balanced approach to MCQ exams by combining efficient assessment of learning with the feedback and improvement opportunities of assessment for learning. As part of a curriculum renewal at Baylor College of Medicine's MD program, the Two-Phase Individual Assessment (TPIA) model was launched within a 4-week preclinical Foundations of Medicine course in August 2023. The TPIA model featured weekly assessments, each consisting of 2 MCQ exams given on the same day with a 4-hour study period in between. Exams were paired, consisting of an equal number of items that addressed the same learning objectives. After the initial exam, students received an individualized feedback report indicating correctly and incorrectly answered objectives. Students applied individualized feedback reports to self-identify and remediate deficiencies in preparation for the second-chance exam. Only the highest score counted toward the final grade. Among 230 medical student participants, significant performance improvements between morning and afternoon exams were observed across the first 3 weekly TPIAs, with mean score increases of 4.93, 5.06, and 10.86. Mean change in performance in week 4 was not significant. Student end-of-course survey responses indicated a strong preference for the TPIA format. Responses highlighted the value of offering individualized feedback, providing unstructured time to address knowledge gaps, and ensuring a corresponding opportunity for improvement. Future research will extend TPIA's implementation in more clinical-oriented courses to validate TPIA's effectiveness and explore its effect on long-term knowledge retention through more cumulative examinations. Integrating TPIA-based approaches may advance educational systems toward developing a culture of feedback and embracing second chances to support learning and professional growth, thereby preparing medical professionals for the demands of 21st-century medicine.