Validation of a Rubric to Assess Physiology Exam Multiple Choice Questions to Create Fair and Equitable Assessments

Abstract

Inequities and achievement gaps exist for students from groups underrepresented in medicine (URiM) and for English language learners (ELLs). This is particularly evident with multiple choice assessments which are prevalent in health professions and medical education. Increasing the linguistic accessibility of test items by simplifying the language used while keeping the content diffculty was perceived as helpful to ELLs in an undergraduate Chemistry course (Lee & Orgill, 2022). Following best practices when writing exam questions provides consistent and valid measures of student comprehension. Across institutions we face similar challenges when forming and assessing multiple choice questions for our physiology exams. Some major flaws include long complex options; using vague terms; negatively structured stems; grammatical cues; word repeats. Lopez (2023) found that 56% items contained at least one flaw. The most common flaws identified were those in which the items were not approximately the same length and item options that did not have parallel grammatical form and structure. In order to address these issues in an objective way, we collectively modified the previously detailed rubric (Lopez, 2023) for identifying common flaws based on the “Item-Writing Guide” from the National Board of Medical Education (NBME®) and one formed in the context of general chemistry by Breakall et al. (2019). Twenty-six questions from physiology exams at three different institutions were evaluated by three faculty using the newly created rubric. After analyzing the questions, we returned as a group and compared our individual ratings, clarified any discrepancies and modified the rubric accordingly. We will conduct statistical analysis to assess our inter-rater reliability score before using this rubric to analyze the full question banks for physiology exams at each institution. This process is not limited to physiology educators since it can be utilized by teachers in any program or discipline. The significance of the work will not only benefit our validation of learning but it will also address specific equity issues by eliminating flaws that provide irrelevant diffculty or benefit test wise students. Furthermore, this rubric is also helpful in generating new questions for future exams, ensuring equitable assessments.