Assessing multiple-choice questions based on language precision and best practices to promote equity in the Dental Physiology course

Abstract

Student assessment in professional schools is conducted mainly through examinations based on multiple-choice questions (MCQs). Although grading this type of assessments saves time, questions that are not well-written may impact students’ performance. Technical flaws in MCQs include those that provide irrelevant difficulty and those that provide an advantage to test-wise examinees. In addition, MCQs with these flaws may disproportionately affect students with weaker undergraduate science backgrounds and those from underrepresented groups including English Language Learners and first-generation college students. Inclusive teaching practices aim to create a level field by removing barriers and providing equal access to students regardless of their background.It is hypothesized that technical flaws in MCQs increase their difficulty. The objectives of this study are: 1) to rate the quality of the MCQs used in a Dental Physiology course at Boston University and 2) to examine the effect of questions with technical flaws on item performance. To measure the performance of specific items, two analyses will be conducted: item difficulty, defined as the percentage of students who choose an item correctly, and item discrimination which refers to the correlation of how well a test taker does on a particular item and their performance on the whole test.An evaluation instrument based on the one developed and validated by Breakall et al. (2019) was employed to identify item writing flaws that add irrelevant difficulty. Examples of item flaws that provide irrelevant difficulty based on the National Board of Medical Examiners (NBME) guidelines are: (a) items with complicated stems and lead-ins that include negative forms, (b) item options that are not written succinctly or include vague terms, (c) numerical data not presented consistently, (d) items that include nonparallel options, or (e) that include “none of the above.” This instrument was used to assess MCQs from a Dental School Physiology exam.The frequency of item flaws indicated that of all items analyzed, 56% items contained at least one flaw. The most common item flaws identified were those where the answer choices were not of approximately the same length (32%), did not have parallel grammatical form and structure (24%) or those that included negative phrasing (16%). In conclusion, this assessment indicates that the MCQs used in this Dental Physiology course have room for improvement. To better understand if the identified flaws affect item performance, exam data provided by Exam Soft Analytics will be analyzed. Based on those results, decisions could be made about modifying MCQs to better serve the needs of our diverse student population.