Plausible Distractors Research Articles

Comparing the Development in Medical Physiology Exam Design in the US and the UK

AIM: To discuss and contrast the recent development of medical physiology assessment between the UK and USA. HYPOTHESIS: The assessment trends set by the USA are regularly imported to the UK, but often the transplants do not take. DISCUSSION: Over the past twenty years educators in the UK have witnessed nominal improvements in medical exam writing policy. Much of this has been driven by American educational advances and theories. The main driver of this one-way trans-Atlantic progression is the larger student numbers in the USA. Until now, the UK has not had a national exam for qualifying newly graduating doctors. The new Medical Licensing Assessment (MLA), which will form the basis of national qualification, will begin to provide such a national test this year (2024) in the UK. Thus far, many USA trends in automated testing have been imported to the UK, but not always successfully. For example, in the early 2000s, nearly all medical schools in the UK instituted extended matching questions (EMQs) [1], but by the late 2000s, nearly all UK medical schools abandoned EMQs [2]. Starting in the late 2000s, single best answer (SBA) formats of 5-option MCQs became a requirement for exam writers. The idea was that high level thinking and judgment would be required to pick the best of the five plausible answers. Again, for post-graduate medical students in the USA, this fit well with diagnostic vignette questions, but for 18-year-old year 1 physiology students in the UK, who were still wrestling with factual knowledge, these judgment exams seemed over-ambitious. Yet, they were enforced upon exam writers. Another policy enforced was that questions could not be negatively phrased, as this has the potential to confuse students under pressure. Although this policy is well-intentioned, negative questions are essential for MCQs directly eliciting student's knowledge on symptoms and other ideas that cannot be reversed. For example, in the item, "What is a symptom of heart failure?" it is nearly impossible to write four plausible distractors, whereas, "All of the following are common symptoms of heart failure EXCEPT" could have "dehydration" as a correct negative answer. Finally, there is a current policy that SBA stems should be phrased so that a student should be able to propose the answer without seeing any of the options. This has been called "the cover test" because if the exam quality team cannot answer the question without seeing the options, the test item is rejected. This means that questions starting, "which of the following" are now rejected immediately. From the student perspective, this policy means that the process of selecting an answer from the list of options is transparent; the answer is recognised rather than weighing proposed options. But this makes it nearly impossible to ask questions about cause-and-effect relations where there are multiple related events such as, "Which of the following is a potential cause of high cardiac preload?" The supposed cover-test version would be, "What causes preload to increase?", which plainly is not a transparent question, as there are so many answers. CONCLUSION: The assessment processes that evolve in the USA are more suitable for the USA context than for the context in the UK. However, the UK educates very well using its home-grown processes. [1] George, S. (2003). Psychiatric Bulletin, 27(6), 230-232. [2] Eijsvogels, T. M., van den Brand, T. L., & Hopman, M. T. (2013). Perspectives on Medical Education, 2, 252-263. None. This is the full abstract presented at the American Physiology Summit 2024 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

Children’s Lexical Anticipation When Listening to Rhymes

ABSTRACT Rhyme awareness is a varied skill emerging consistently only in late preschool, yet children respond to rhythmic and phonological patterns to spontaneously complete rhymes in everyday settings. Our study replicates and extends previous work using a modified preferential looking task to test whether preschoolers can efficiently use rhyme to anticipate spoken words. Three- to 4-year-olds (n = 30) listened to 24 simple three-line rhymes as we measured their looking patterns during the two preceding lines of a stanza and a silent anticipatory pause before the final word of the rhyme was heard. Children looked more toward illustrations of the target word that could complete each rhyme relative to two plausible distractors (p < .001) indicating that they can use rhyme to anticipate upcoming words. These findings challenge us to think more deeply about how we measure rhyme awareness and what sources of information can be recruited by young children to achieve efficient comprehension.

The Advantages of Five-Option Multiple-Choice Items in Classroom Tests of Student Mastery

The effectiveness of multiple-choice (MC) items depends on the quality of the response options—particularly how well the incorrect options (“distractors”) attract students who have incomplete knowledge. It is often contended that test-writers are unable to devise more than two plausible distractors for most MC items, and that the effort needed to do so is not worthwhile in terms of the items’ psychometric qualities. To test these contentions, I analyzed students’ performance on 545 MC items across six science courses that I have taught over the past decade. Each MC item contained four distractors, and the dataset included more than 19,000 individual responses. All four distractors were deemed plausible in one-third of the items, and three distractors were plausible in another third. Each increase in plausible distractor led to an average of a 13% increase in item difficulty. Moreover, an increase in plausible distractors led to a significant increase in the discriminability of the items, with a leveling off by the fourth distractor. These results suggest that—at least for teachers writing tests to assess mastery of course content—it may be worthwhile to eschew recent skepticism and continue to attempt to write MC items with three or four distractors.

Multiple-Choice Item Distractor Development Using Topic Modeling Approaches.

Writing a high-quality, multiple-choice test item is a complex process. Creating plausible but incorrect options for each item poses significant challenges for the content specialist because this task is often undertaken without implementing a systematic method. In the current study, we describe and demonstrate a systematic method for creating plausible but incorrect options, also called distractors, based on students’ misconceptions. These misconceptions are extracted from the labeled written responses. One thousand five hundred and fifteen written responses from an existing constructed-response item in Biology from Grade 10 students were used to demonstrate the method. Using a topic modeling procedure commonly used with machine learning and natural language processing called latent dirichlet allocation, 22 plausible misconceptions from students’ written responses were identified and used to produce a list of plausible distractors based on students’ responses. These distractors, in turn, were used as part of new multiple-choice items. Implications for item development are discussed.

Creating assessments as an active learning strategy: what are students’ perceptions? A mixed methods study

ABSTRACTBackground: Teaching students how to create assessments, such as those involving multiple-choice questions (MCQs), has the potential to be a useful active learning strategy. In order to optimize students’ learning, it is essential to understand how they engage with such activities.Objective: To explore medical students’ perceptions of how completing rigorous MCQ training and subsequently writing MCQs affects their learning.Design: In this mixed methods exploratory qualitative study, eighteen second-year medical students, trained in MCQ-writing best practices, collaboratively generated a question bank. Subsequently, the authors conducted focus groups with eight students to probe impressions of the process and the effect on learning. Responses partially informed a survey consisting of open-ended and Likert rating scale questions that the remaining ten students completed. Focus group and survey data from the eighteen participants were iteratively coded and categorized into themes related to perceptions of training and of collaborative MCQ writing.Results: Medical students felt that training in MCQ construction affected their appreciation for MCQ examinations and their test-taking strategy. They perceived that writing MCQs required more problem-solving and content-integration compared to their preferred study strategies. Specifically, generating plausible distractors required the most critical reasoning to make subtle distinctions between diagnoses and treatments. Additionally, collaborating with other students was beneficial in providing exposure to different learning and question-writing approaches.Conclusions: Completing MCQ-writing training increases appreciation for MCQ assessments. Writing MCQs requires medical students to make conceptual connections, distinguish between diagnostic and therapeutic options, and learn from colleagues, but requires extensive time and knowledge base.

A psychometric analysis of a newly developed summative, multiple choice question assessment adapted from Canada to a Middle Eastern context

IntroductionAccreditation necessitates that assessment methods reflect the standards established by the accrediting body. The process of adapting assessments to a new context can present unique challenges with uncertainty around psychometric defensibility of the adapted exam. MethodsA psychometric analysis of a summative multiple-choice-question (MCQ) assessment, adapted from Canada, for graduating pharmacy students from a Canadian accredited program in Qatar was conducted. Rates of difficult items, item discrimination measured by point biserial correlation (rpb), and non-functioning distractors (NFDs) were calculated to identify deficiencies and challenges with an adapted exam. Challenges encountered throughout the adaption process and recommendations were documented. ResultsOverall score of a 90-item, four option, MCQ exam ranged from 46.7% to 78.9% (mean of 61.9%). For difficulty, there were 17 items with less than 30% of students answering correctly, while 29 items had unacceptable or poor discrimination (rpb < 0.1). NFDs occurred in 78 items with 49 containing at least two NFDs. Discussion and conclusionsDifficulty of the exam was deemed acceptable yet discriminator ability requires improvement. The high frequency of questions with NFDs suggests that faculty have difficulty developing plausible distractors for an adapted MCQ exam. This could be due to a lack of training or requirement for inclusion of too many distractor options. While it is feasible to implement an assessment adapted from a different learning environment, measures need to be taken to improve psychometric defensibility. The high number of questions with NFDs indicates that the current method of exam development does not encourage the incorporation of functional distractors.

Three, four, and none of the above options in multiple-choice items

High-quality multiple-choice (MC) items are essential for creating efficient, valid assessments. Haladyna, Downing, and Rodriguez (2002) suggested that using plausible distractors is crucial to achieving this goal, although distractor creation can be time-consuming and challenging. Haladyna et al. (2002) provided two related test development guidelines: #18, “Write as many plausible distractors as you can,” and #25, “Use carefully None of the above.” This research aims to test the impact of these two guidelines on item difficulty (p), item discrimination (r), and test reliability for mathematics items empirically. The research findings have revealed that item discrimination and test reliability were not statistically different across MC items with four options, three options, and NOTA options while the means of item difficulty of four-option MC items was not statistically different from those of three-option and NOTA-option MC items. However, the mean of item difficulty of NOTA-option MC items was statistically lower than those of three-option.

Use of Item Analysis to Improve the Quality of Single Best Answer Multiple Choice Question in Summative Assessment of Undergraduate Medical Students in Malaysia.

Background: Single best answer (SBA) as multiple-choice items are often advantageous to use for its reliability and validity. However, SBA requires good number of plausible distractors to achieve reliability. Apart from psychometric evaluation of assessment it is important to perform item analysis to improve quality of items by analysing difficulty index (DIF I), discrimination index (DI) and distractor efficiency (DE) based on number of non-functional distractors (NFD). Objective: To evaluate quality of SBA items administered in professional examination to apply corrective measures determined by DIF I, DI and DE using students’ assessment score. Method: An evaluation of post summative assessment (professional examination) of SBA items as part of psychometric assessment is performed after 86 weeks of teaching in preclinical phase of MD program. 40 SBA items and 160 distractors inclusive of key were assessed using item analysis. 136 students’ score of SBA was analysed for mean and standard deviation, DIF I, DI and DE using MS Excel 2007. Unpaired t -test was applied to determine DE in relation to DIF I and DI with level of significance. Item-total correlation (r) and internal consistency by Cronbach’s alpha and parallel-form method was also computed. Result: 15 items had DIF I=.31-.61 and 25 items had DIF I ≤.30 or ≥0.61). 26 items had DI = 0.15 - ≥.25 compared to 14 items with DI (≤0.15). There were 26 (65%) items with 1-3 NFD and 14 (35%) items without any NFD. 39 (32.50%) distractors were with choice frequency=0. Overall mean DE was 65.8% and NFD was 49 (40.5%). DE in relation to DIF I and DI were statistically significant with p = .010 and .020 respectively. Item-total correlation for most items was <.3. Internal consistency by Cronbach’s alpha in SBA test 1 and 2 was .51 and .41 respectively and constancy by parallel-form method was .57 between SBA test 1 and 2. Conclusion: The high frequency of difficult or easy items and moderate to poor discrimination suggest the need of items corrective measure. Increased number of NFD and low DE in this study indicates difficulty of teaching faculty in developing plausible distractors for SBA questions. This has been reflected in poor reliability established by alpha. Item analysis result emphasizes the need of evaluation to provide feedback and to improve quality of SBA items in assessment.

Developing, evaluating, and refining an automatic generator of diagnostic multiple choice cloze questions to assess children's comprehension while reading

AbstractWe describe the development, pilot-testing, refinement, and four evaluations of Diagnostic Question Generator (DQGen), which automatically generates multiple choice cloze (fill-in-the-blank) questions to test children's comprehension while reading a given text. Unlike previous methods, DQGen tests comprehension not only of an individual sentence but of the context preceding it. To test different aspects of comprehension, DQGen generates three types of distractors: ungrammatical distractors test syntax; nonsensical distractors test semantics; and locally plausible distractors test inter-sentential processing.(1)A pilot study of DQGen 2012 evaluated its overall questions and individual distractors, guiding its refinement into DQGen 2014.(2)Twenty-four elementary students generated 200 responses to multiple choice cloze questions that DQGen 2014 generated from forty-eight stories. In 130 of the responses, the child chose the correct answer. We define thedistractivenessof a distractor as the frequency with which students choose it over the correct answer. The incorrect responses were consistent with expected distractiveness: twenty-seven were plausible, twenty-two were nonsensical, fourteen were ungrammatical, and seven were null.(3)To compare DQGen 2014 against DQGen 2012, five human judges categorized candidate choices without knowing their intended type or whether they were the correct answer or a distractor generated by DQGen 2012 or DQGen 2014. The percentage of distractors categorized as their intended type was significantly higher for DQGen 2014.(4)We evaluated DQGen 2014 against human performance based on 1,486 similarly blind categorizations by twenty-seven judges of sixteen correct answers, forty-eight distractors generated by DQGen 2014, and 504 distractors authored by twenty-one humans. Surprisingly, DQGen 2014 did significantly better than humans at generating ungrammatical distractors and marginally better than humans at generating nonsensical distractors, albeit slightly worse at generating plausible distractors. Moreover, vetting DQGen 2014's output and writing distractors only when necessary would halve the time to write them all, and produce higher quality distractors.

Validity and Reliability of Scores Obtained on Multiple-Choice Questions: Why Functioning Distractors Matter

Purpose Plausible distractors are important for accurate measurement of knowledge via multiple-choice questions (MCQs). This study demonstrates the impact of higher distractor functioning on validity and reliability of scores obtained on MCQs. Methods Free-response (FR) and MCQ versions of a neurohistology practice exam were given to four cohorts of Year 1 medical students. Consistently non-functioning multiple-choice distractors (&lt;5% selection frequency) were replaced with those developed from incorrect responses on FR version of the items, followed by administration of the revised MCQ version to subsequent two cohorts. Validity was assessed by comparing an index of expected MCQ difficulty with an index of observed MCQ difficulty, while reliability was assessed via Cronbach’s alpha coefficient before and after replacement of consistently non-functioning distractors. Result Pre-intervention, effect size (Cohen’s d) of the difference between mean expected and observed MCQ difficulty indices was noted to be 0.4 – 0.59. Post-intervention, this difference reduced to 0.15 along with an increase in Cronbach’s alpha coefficient of scores obtained on MCQ version of the exam. Conclusion Multiple-choice distractors developed from incorrect responses on free-response version of the items enhance the validity and reliability of scores obtained on MCQs.

Using Automatic Item Generation to Improve the Quality of MCQ Distractors

ABSTRACTConstruct: Automatic item generation (AIG) is an alternative method for producing large numbers of test items that integrate cognitive modeling with computer technology to systematically generate multiple-choice questions (MCQs). The purpose of our study is to describe and validate a method of generating plausible but incorrect distractors. Initial applications of AIG demonstrated its effectiveness in producing test items. However, expert review of the initial items identified a key limitation where the generation of implausible incorrect options, or distractors, might limit the applicability of items in real testing situations. Background: Medical educators require development of test items in large quantities to facilitate the continual assessment of student knowledge. Traditional item development processes are time-consuming and resource intensive. Studies have validated the quality of generated items through content expert review. However, no study has yet documented how generated items perform in a test administration. Moreover, no study has yet to validate AIG through student responses to generated test items. Approach: To validate our refined AIG method in generating plausible distractors, we collected psychometric evidence from a field test of the generated test items. A three-step process was used to generate test items in the area of jaundice. At least 455 Canadian and international medical graduates responded to each of the 13 generated items embedded in a high-stake exam administration. Item difficulty, discrimination, and index of discrimination estimates were calculated for the correct option as well as each distractor. Results: Item analysis results for the correct options suggest that the generated items measured candidate performances across a range of ability levels while providing a consistent level of discrimination for each item. Results for the distractors reveal that the generated items differentiated the low- from the high-performing candidates. Conclusions: Previous research on AIG highlighted how this item development method can be used to produce high-quality stems and correct options for MCQ exams. The purpose of the current study was to describe, illustrate, and evaluate a method for modeling plausible but incorrect options. Evidence provided in this study demonstrates that AIG can produce psychometrically sound test items. More important, by adapting the distractors to match the unique features presented in the stem and correct option, the generation of MCQs using automated procedure has the potential to produce plausible distractors and yield large numbers of high-quality items for medical education.

Distratores e chaves em um cloze lexical de escolha múltipla de nível avançado: a oportunidade de considerar o julgamento de falantes nativos

Na pesquisa aqui apresentada, dois grupos diferentes de falantes nativos com elevado grau de conhecimento da língua (n=37) realizaram 2 cloze lexicais de múltipla escolha destinados a estudantes de nível C2 e extraídos do exame de certificação do italiano como língua estrangeira CILS (Università per Stranieri di Siena). As respostas divergentes em relação às previstas pelos responsáveis pela certificação foram analisadas. Da análise emerge a plausibilidade de alguns distratores. Nas conclusões discutimos a necessidade de os responsáveis pela certificação considerar o julgamento de falantes nativos com alto grau de conhecimento linguístico como forma de validar o teste.

Motivation for and Development of a Standardized Introductory Meteorology Assessment Exam

Abstract Education research has shown that there is often a disconnect between what instructors teach and what students actually comprehend. Much of this disconnect stems from students’ previous conceptions of the subject that often remain steadfast despite instruction. The field of meteorology is particularly susceptible to misconceptions as a result of the years of personal experience students have with weather before instruction. Consequently, it is often challenging for students to accurately integrate course material with their observations and personal explanations. A longitudinal assessment exam of the meteorology program at the U.S. Air Force Academy revealed that misconceptions of fundamental, introductory content can propagate through years of instruction, potentially impeding deeper understanding of advanced topics and hindering attainment of professional certifications. Thus, it is clear that such misconceptions must be identified and corrected early. This manuscript describes the development of the Fundamentals in Meteorology Inventory (FMI), a multiple-choice assessment exam designed to identify the common misconceptions of fundamental topics covered in introductory meteorology courses. In developing the FMI, care was taken to avoid complex vocabulary and to include plausible distractors identified by meteorology faculty members. Question topics include clouds and precipitation, wind, fronts and air masses, temperature, stability, severe weather, and climate. Applications of the exam for the meteorology community are discussed, including identifying common meteorology misconceptions, assessing student understanding, measuring teaching effectiveness, and diagnosing areas for improvement in introductory meteorology courses. Future work to be completed to ensure the efficacy of the FMI will also be acknowledged.

Making concepts of medical biochemistry by formulating distractors of multiple choice questions: growing mighty oaks from small acorns and #8217;

Background: Multiple choice questions (MCQs) have routinely been used for formative evaluation and selection purposes. Enormous efforts are required to produce MCQs with plausible distractors, which test the understanding of concepts and problem solving abilities, without being trivial or giving clues to the correct answer. A thorough knowledge of the concerned topic is a prerequisite for the setter. Therefore in the present study, we observed the effect of formulating distractors of the given MCQs on understanding of a particular topic (“enzymes”) of medical biochemistry in undergraduate medical students. Materials and Methods: This study was carried out in the Department of Biochemistry in Jawaharlal Institute of Postgraduate Medical Education and Research, Puducherry, India, and involved 135 first year medical students. Students were divided into three groups based on their internal assessment (IA) performance: High performers with >75% IA (n = 44), medium performers with IA 60-75% (n = 48), and low performers with IA 0.05) compared with the first post-test. Although there was no statistical significance between the gain obtained by all the three groups, appreciable gain was observed for medium performers (+1.0698) and low performers (+0.755). Conclusion: Formulation of distracters of MCQs for a given stem enhanced the comprehension and performance of medical undergraduate students in medical biochemistry.

Multiple-Choice Item Generator for Narrative and Declarative Texts

 Abstract—Multiple Choice Item Generator is a research focused on generating grammatically and semantically correct questions and generating plausible distractors. The system's framework has three main modules: pre-processing, question generation and distractor selection. Pre-processing module implements Hobbs Algorithm for anaphora resolution and direct to indirect speech conversion for handling quoted statements. In generating questions, question overgeneration and ranking framework of Heilman and Smith is used. The distractor selection module uses collocation extraction and Wordnet-based methods such as Lin's semantic similarity measure, hypernyms, hyponyms and maximal bipartite graph matching .

Distractores en preguntas de opción múltiple para estudiantes de medicina: ¿cuál es su comportamiento en un examen sumativo de altas consecuencias?

IntroductionMultiple-choice question (MCQ) exams are the most frequently used tool for knowledge assessment of medical students. It has been shown that three options in each item are enough; however, many tests in our country still use questions with five options. The study of non-functional distractors (NFD) is needed in order to improve the quality of our assessment instruments. ObjectiveTo identify NFD and their behavior in a high-stakes summative exam in medical students. MethodA psychometric analysis was performed of the Theoretical Professional Exam at UNAM Faculty of Medicine, the 2008 version. Difficulty, discrimination and point-biserial correlation were calculated for each item and its four distractors. The frequency of items with zero to four NFD was obtained, and the differences of NFD per item and their psychometric characteristics were assessed. The psychometric behavior of the whole exam was compared to a version where items with four NFD were removed. ResultsThe test had 420 MCQ items with five options, and was answered by 882 students. The instrument had a Cronbach’s alpha of 0.93. Of the 1 680 distractors evaluated, only 788 (46.9%) were functional. More than two-thirds of the total of items had two or more NFD. There was an average of 2.12 ± 0.99 NFD per item, the majority of which were chosen by less than 5% of the test takers. With higher quantity of NFD, the items’ difficulty index was higher and the discrimination index was lower. ConclusionsItems with two functional distractors were the majority in the exam. Two plausible distractors are a reasonable alternative for item development, maintaining or improving the reliability and psychometric profile of these types of tests.

Evaluating the quality of medical multiple‐choice items created with automated processes

Computerised assessment raises formidable challenges because it requires large numbers of test items. Automatic item generation (AIG) can help address this test development problem because it yields large numbers of new items both quickly and efficiently. To date, however, the quality of the items produced using a generative approach has not been evaluated. The purpose of this study was to determine whether automatic processes yield items that meet standards of quality that are appropriate for medical testing. Quality was evaluated firstly by subjecting items created using both AIG and traditional processes to rating by a four-member expert medical panel using indicators of multiple-choice item quality, and secondly by asking the panellists to identify which items were developed using AIG in a blind review. Fifteen items from the domain of therapeutics were created in three different experimental test development conditions. The first 15 items were created by content specialists using traditional test development methods (Group 1 Traditional). The second 15 items were created by the same content specialists using AIG methods (Group 1 AIG). The third 15 items were created by a new group of content specialists using traditional methods (Group 2 Traditional). These 45 items were then evaluated for quality by a four-member panel of medical experts and were subsequently categorised as either Traditional or AIG items. Three outcomes were reported: (i) the items produced using traditional and AIG processes were comparable on seven of eight indicators of multiple-choice item quality; (ii) AIG items can be differentiated from Traditional items by the quality of their distractors, and (iii) the overall predictive accuracy of the four expert medical panellists was 42%. Items generated by AIG methods are, for the most part, equivalent to traditionally developed items from the perspective of expert medical reviewers. While the AIG method produced comparatively fewer plausible distractors than the traditional method, medical experts cannot consistently distinguish AIG items from traditionally developed items in a blind review.

Rarely selected distractors in high stakes medical multiple-choice examinations and their recognition by item authors: a simulation and survey

BackgroundMany medical exams use 5 options for multiple choice questions (MCQs), although the literature suggests that 3 options are optimal. Previous studies on this topic have often been based on non-medical examinations, so we sought to analyse rarely selected, 'non-functional' distractors (NF-D) in high stakes medical examinations, and their detection by item authors as well as psychometric changes resulting from a reduction in the number of options.MethodsBased on Swiss Federal MCQ examinations from 2005-2007, the frequency of NF-D (selected by <1% or <5% of the candidates) was calculated. Distractors that were chosen the least or second least were identified and candidates who chose them were allocated to the remaining options using two extreme assumptions about their hypothetical behaviour: In case rarely selected distractors were eliminated, candidates could randomly choose another option - or purposively choose the correct answer, from which they had originally been distracted. In a second step, 37 experts were asked to mark the least plausible options. The consequences of a reduction from 4 to 3 or 2 distractors - based on item statistics or on the experts' ratings - with respect to difficulty, discrimination and reliability were modelled.ResultsAbout 70% of the 5-option-items had at least 1 NF-D selected by <1% of the candidates (97% for NF-Ds selected by <5%). Only a reduction to 2 distractors and assuming that candidates would switch to the correct answer in the absence of a 'non-functional' distractor led to relevant differences in reliability and difficulty (and to a lesser degree discrimination). The experts' ratings resulted in slightly greater changes compared to the statistical approach.ConclusionsBased on item statistics and/or an expert panel's recommendation, the choice of a varying number of 3-4 (or partly 2) plausible distractors could be performed without marked deteriorations in psychometric characteristics.

An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis

BackgroundFour- or five-option multiple choice questions (MCQs) are the standard in health-science disciplines, both on certification-level examinations and on in-house developed tests. Previous research has shown, however, that few MCQs have three or four functioning distractors. The purpose of this study was to investigate non-functioning distractors in teacher-developed tests in one nursing program in an English-language university in Hong Kong.MethodsUsing item-analysis data, we assessed the proportion of non-functioning distractors on a sample of seven test papers administered to undergraduate nursing students. A total of 514 items were reviewed, including 2056 options (1542 distractors and 514 correct responses). Non-functioning options were defined as ones that were chosen by fewer than 5% of examinees and those with a positive option discrimination statistic.ResultsThe proportion of items containing 0, 1, 2, and 3 functioning distractors was 12.3%, 34.8%, 39.1%, and 13.8% respectively. Overall, items contained an average of 1.54 (SD = 0.88) functioning distractors. Only 52.2% (n = 805) of all distractors were functioning effectively and 10.2% (n = 158) had a choice frequency of 0. Items with more functioning distractors were more difficult and more discriminating.ConclusionThe low frequency of items with three functioning distractors in the four-option items in this study suggests that teachers have difficulty developing plausible distractors for most MCQs. Test items should consist of as many options as is feasible given the item content and the number of plausible distractors; in most cases this would be three. Item analysis results can be used to identify and remove non-functioning distractors from MCQs that have been used in previous tests.

Distractor plausibility and criterion placement in recognition

To set an optimal decision criterion on a test of recognition, a subject must estimate the degree to which they can discriminate previously studied from unstudied stimuli. To do so accurately, the subject must assess not only their mastery of the material but also the extent to which the distractors yield mnemonic evidence that makes them difficult to discriminate from studied items. In these experiments, we manipulated the degree to which the distractor set overlapped semantically or categorically with the previously learned stimuli, and examined the effects on criterion placement. When plausibility was manipulated by semantic membership, subjects who took tests with more plausible distractors set a higher criterion; for all other manipulations of plausibility, there were no between-subject effects of plausibility on criterion placement. However, over multiple test opportunities, subjects increased their criterion when the tests became more difficult, but they did not lower their criterion when the test became easier. This asymmetry obtained with picture and word categories, and suggests that online monitoring of recognition performance modulates the criterion shift.