Incorrect Options Research Articles

Identifying student profiles in a digital mental rotation task: insights from the 2017 NAEP math assessment

Mental rotation (MR), a key aspect of spatial reasoning, is highly predictive of success in STEM fields. This study analyzed strategies employed by 27,600 eighth-grade students during a digital MR task from the 2017 National Assessment of Educational Progress (NAEP) in mathematics. Utilizing K-means cluster analysis to categorize behavioral and performance patterns, we identified four distinct profiles: Cognitive Offloaders (15% of the sample), Internal Visualizers (55%), External Visualizers (5%), and Non-Triers (25%). Cognitive Offloaders, skilled at minimizing cognitive load by eliminating incorrect options, demonstrated the highest MR accuracy rates at 45%. Internal Visualizers, relying less on digital tools and more on mental strategies, achieved robust performance with an average score of 38%. External Visualizers, despite their extensive use of assistive tools and greater time investment, scored an average of 36%. Non-Triers showed minimal engagement and correspondingly the lowest performance, averaging 29%. These findings not only underscore the diverse strategies students adopt in solving MR tasks but also emphasize the need for educational strategies that are tailored to accommodate different cognitive styles. By integrating MR training into the curriculum and enhancing teacher preparedness to support diverse learning needs, this study advocates for educational reforms to promote equitable outcomes in mathematics and broader STEM fields.

MedExpQA: Multilingual benchmarking of Large Language Models for Medical Question Answering

Large Language Models (LLMs) have the potential of facilitating the development of Artificial Intelligence technology to assist medical experts for interactive decision support. This potential has been illustrated by the state-of-the-art performance obtained by LLMs in Medical Question Answering, with striking results such as passing marks in licensing medical exams. However, while impressive, the required quality bar for medical applications remains far from being achieved. Currently, LLMs remain challenged by outdated knowledge and by their tendency to generate hallucinated content. Furthermore, most benchmarks to assess medical knowledge lack reference gold explanations which means that it is not possible to evaluate the reasoning of LLMs predictions. Finally, the situation is particularly grim if we consider benchmarking LLMs for languages other than English which remains, as far as we know, a totally neglected topic. In order to address these shortcomings, in this paper we present MedExpQA, the first multilingual benchmark based on medical exams to evaluate LLMs in Medical Question Answering. To the best of our knowledge, MedExpQA includes for the first time reference gold explanations, written by medical doctors, of the correct and incorrect options in the exams. Comprehensive multilingual experimentation using both the gold reference explanations and Retrieval Augmented Generation (RAG) approaches show that performance of LLMs, with best results around 75 accuracy for English, still has large room for improvement, especially for languages other than English, for which accuracy drops 10 points. Therefore, despite using state-of-the-art RAG methods, our results also demonstrate the difficulty of obtaining and integrating readily available medical knowledge that may positively impact results on downstream evaluations for Medical Question Answering. Data, code, and fine-tuned models will be made publicly available.11https://huggingface.co/datasets/HiTZ/MedExpQA.

Unaided Visual Inspection for Assessment of Penile Curvature in the Clinical Setting of Hypospadias Surgery: Survey of Members of Society of Pediatric Urology (India).

To compare the accuracy of unaided visual inspection (UVI) to Software App measurement (SAM) of penile curvature (PC) during hypospadias surgery. Seven clinical pictures of PC (15°-60°) taken during hypospadias repair were shared with 300 members of the Society of Pediatric Urology (India). The respondents were asked to assess the angles by UVI and indicate their preferred correction method of that PC. For each picture, the angles of curvature estimated by UVI were compared with the objective angle measured using an app (SAM), which was considered an accurate estimation. Statistical analysis was done using software; P<0.05 was considered as statistically significant. Ninety-one of 101 (90%) respondents preferred UVI to measure PC during hypospadias surgery. For 6/7 pictures, <40% of participants estimated the angle correctly by UVI (P < 0.001), with the difference in estimation being 3.6°-14.9°. For pictures with PC >30°, the error in UVI estimation was >10°, with no correlation between the accuracy of UVI estimate and surgeon experience. A significant proportion of surgeons chose the incorrect option for PC correction, which was the lowest (69%) for PC 35.8°. Most surgeons preferred UVI to assess PC; UVI is an erroneous technique to measure PC angle, especially in the PC range 30°-60°, where the error was >10°. Most errors were an underestimation of the PC, irrespective of surgeon experience. There was a significant error in the choice of technique for PC correction for a PC of 35°. These results strongly support the objective assessment of PC using SAM during hypospadias repair.

Using OpenAI GPT to Generate Reading Comprehension Items

AbstractThe purpose of this study is to introduce and evaluate a method for generating reading comprehension items using template‐based automatic item generation. To begin, we describe a new model for generating reading comprehension items called the text analysis cognitive model assessing inferential skills across different reading passages. Next, the text analysis cognitive model is used to generate reading comprehension items where examinees are required to read a passage and identify the irrelevant sentence. The sentences for the generated passages were created using OpenAI GPT‐3.5. Finally, the quality of the generated items was evaluated. The generated items were reviewed by three subject‐matter experts. The generated items were also administered to a sample of 1,607 Grade‐8 students. The correct options for the generated items produced a similar level of difficulty and yielded strong discrimination power while the incorrect options served as effective distractors. Implications of augmented intelligence for item development are discussed.

The Features of Plausible but Incorrect Options: Distractor Plausibility in Synonym-Based Vocabulary Tests

A better understanding of how distractor features influence the plausibility of distractors is essential for an efficient multiple-choice (MC) item construction in educational assessment. The plausibility of distractors has a major influence on the psychometric characteristics of MC items. Our analysis utilizes the nominal categories model to investigate German fourth graders' ( N = 924) selection of response options in a German MC Vocabulary test. We used principles from cognitive psychology to identify relevant option features capturing the option’s potential to distract students from the correct answer. The results show that only a few option characteristics explain option choice behavior to a large extent. Options with distracting features (i.e., semantic relatedness and orthographic similarity) increase the item difficulty and discrimination, whereas distractors that are less synonym than the attractor decrease item discrimination. Implications for test score interpretations and item construction guidelines are highlighted.

The Nightmare Island: Representations of St. Barthélemy in Swedish Novels

The Nightmare Island: Representations of St. Barthélemy in Swedish Novels

Public perception of household risks for pediatric burn injuries and assessment of management readiness

IntroductionChildren are uniquely vulnerable to injury because of near-complete dependence on caregivers. Unintentional injury is leading cause of death in children under the age of 14. Burns are one of the leading causes of accidental and preventable household injuries, with scald burns most common in younger children and flame burns in older ones. Education is a key tool to address burn prevention, but unfortunately these injuries persist. Critically, there is a paucity of literature investigating adult comprehension with respect to potential risks of household burns. To date, no study has been performed to assess management readiness for these types of injuries without seeking medical care. MethodsQualtrics™ surveys were distributed to laypersons via Amazon Mechanical Turk. Demographics were self-reported. The survey was divided into two parts, management knowledge, and risk identification. The management part involved a photograph of a first-degree pediatric burn injury and required identification of the degree of injury and three potential initial managements. The risk-identification section required correctly identifying the most common mechanisms of burn injury for different age groups followed by general identification of 20 household burn risks. Survey responses were analyzed using two-tailed Student's t-tests and chi-square analyses, univariate and multivariate analysis, and linear regression. ResultsOf the 467 respondents, the mean age was 36.57 years, and was 59.7% (279) male. Only 3.2% of respondents were able to correctly identify all 20 potential risks listed in our survey. Additionally, only 4.5% of respondents correctly identified all three appropriate initial management options (cool water, sterile gauze, and over-the-counter analgesics) without misidentifying incorrect options. However, 56.1% of respondents were able to select at least one correct management option. For image-based injury classification, the most common response was incorrectly second-degree with 216 responses (42.2%) and the second-most common response was correctly first-degree with 146 responses (31.3%). Most respondents claimed they would not seek medical attention for the injury presented in the photograph (77.7%). When comparing the responses of individuals with children to those without, there were no statistically significant differences in ability to assess household risks for pediatric burns. For the entire population of respondents, the mean score for correctly identifying risks was 38%. ConclusionThis study revealed a significant gap in public awareness of household risks for pediatric burns. Furthermore, while most individuals would not seek medical care for a first-degree pediatric burn injury, they were readily available to identify proper initial management methods. This gap in knowledge and understanding of household pediatric burn injuries should be addressed with increased burn injury prevention education initiatives and more parental counseling opportunities.

The presence of irrelevant alternatives paradoxically increases confidence in perceptual decisions

Confidence in perceptual decisions is thought to reflect the probability of being correct. According to this view, confidence should be unaffected or minimally reduced by the presence of irrelevant alternatives. To test this prediction, we designed five experiments. In Experiment 1, participants had to identify the largest geometrical shape among two or three alternatives. In the three-alternative condition, one of the shapes was much smaller than the other two, being a clearly incorrect option. Counter-intuitively, confidence was higher when the irrelevant alternative was present, evidencing that confidence construction is more complex than previously thought. Four computational models were tested, only one of them accounting for the results. This model predicts that confidence increases monotonically with the number of irrelevant alternatives, a prediction we tested in Experiment 2. In Experiment 3, we evaluated whether this effect replicated in a categorical task, but we did not find supporting evidence. Experiments 4 and 5 allowed us to discard stimuli presentation time as a factor driving the effect. Our findings suggest that confidence models cannot ignore the effect of multiple, possibly irrelevant alternatives to build a thorough understanding of confidence.

Three Sources of Validation Evidence Needed to Evaluate the Quality of Generated Test Items for Medical Licensure

Issue: Automatic item generation is a method for creating medical items using an automated, technological solution. Automatic item generation is a contemporary method that can scale the item development process for production of large numbers of new items, support building of multiple forms, and allow rapid responses to changing medical content guidelines and threats to test security. The purpose of this analysis is to describe three sources of validation evidence that are required when producing high-quality medical licensure test items to ensure evidence for valid test score inferences, using the automatic item generation methodology for test development. Evidence: Generated items are used to make inferences about examinees’ medical knowledge, skills, and competencies. We present three sources of evidence required to evaluate the quality of the generated items that is necessary to ensure the generated items measure the intended knowledge, skills, and competencies. The sources of evidence we present here relate to the item definition, the item development process, and the item quality review. An item is defined as an explicit set of properties that include the parameters, constraints, and instructions used to elicit a response from the examinee. This definition allows for a critique of the input used for automatic item generation. The item development process is evaluated using a validation table, whose purpose is to support verification of the assumptions related to model specification made by the subject-matter expert. This table provides a succinct summary of the content and constraints that were used to create new items. The item quality review is used to evaluate the statistical quality of the generated items, which often focuses on the difficulty and the discrimination of the correct and incorrect options. Implications: Automatic item generation is an increasingly popular item development method. The generated items from this process must be bolstered by evidence to ensure the items measure the intended knowledge, skills, and competencies. The purpose of this analysis is to describe these sources of evidence that can be used to evaluate the quality of the generated items. The important role of medical expertise in the development and evaluation of the generated items is highlighted as a crucial requirement for producing validation evidence.

Assessing Repetitive Trials in Serious Games

Players in serious games may often need multiple trials for correctly completing a game task. Therefore, the number of attempts should be reflected in the score. This article presents three computational score models that take into account the number of attempts that a player makes to be successful. The models, which are extensions of test theory, quantify the random contributions to the scores that need to be removed. They also describe the influence of prior knowledge used for elimination of incorrect options, and take into account that the decision options in a node may not be equally plausible. In a series of simulation studies the score outcomes of the models are compared under various conditions. Results show that the number of trials cannot be ignored as they have a strong impact on the performance scores to be assigned. Neglecting the number of trials leads to inaccurate scores that significantly overrate the observed performances, occasionally up to 100% or even more. The effects depend on the number of decision options, the presence of obvious, correct or incorrect options given the player&amp;acute;s knowledge level and, to a lesser extent, different plausibility levels of the options to decide upon. The practical feasibility is high, because a simple score formula largely solves the problem.

New Modification of Smell Identification Test for the Detection of Malingerers: A Pilot Experimental Study.

Background: A major problem with the University of Pennsylvania Smell Identification Test (UPSIT) is its poor sensitivity for malingering detection in a group of people familiar with the test mechanism. This study aimed to evaluate the modification of UPSIT to detect anosmia malingering.Methods: This was a pilot experimental study conducted in 2019 in Tehran. The participants were 60 healthy subjects classified into two groups of 30 people. The first group was requested to deliberately feign a negative result on the Iranian version of UPSIT, Iran Smell Identification Test (ISIT) (malingering group). The second group consisted of participants, who did not scratch the odorant part of ISIT during the tests (anosmia group). ISIT was modified in two steps. At each step, one incorrect option was deleted from the available choices. The number of each group’s answers, altered away from the correct choice, was then calculated and compared.Results: The coached malingering group participants were able to feign anosmia in the original ISIT exam. In the modified ISIT, the number of answers changed from correct to wrong during the second stage (from three available choices to two choices) was significantly higher in the anosmia group (P<0.001). In the ROC analysis, the area under the curve was 0.92 (P<0.001). The cut-off of 4.5 for this test showed 93% sensitivity, 82% specificity, and 90% PPV and NPV.Conclusion: The ISIT is not capable of detecting malingering in the coached participants, yet by deleting the choices step-by-step, the sensitivity and specificity of the test increased.

Diverse Distractor Generation for Constructing High-Quality Multiple Choice Questions

Distractor generation task aims to generate incorrect options (i.e., distractors) for multiple choice questions from an article.Existing methods for this task often utilize a standard encoder-decoder framework. However, these methods often tend to generate semantically similar distractors, since the same article representations are used to generate different distractors. Multiple generated distractors with similar semantics are considered equivalent. Because the correct answer is unique, students can eliminate these distractors even without reading the article. In this paper, we propose a multi-selector generation network (MSG-Net) that generates distractors with rich semantics based on different sentences in an article. MSG-Net adopts a multi-selector mechanism to select multiple different sentences in an article that are useful to generate diverse distractors. Specifically, a question-aware and answer-aware mechanism are introduced to assist in selecting useful key sentences, where each key sentence is coherent with the question and not equivalent to the answer. MSG-Net can generate diverse distractors based on each selected key sentence with different semantics. Extensive experiments on the RACE dataset and Cosmos QA dataset show that the proposed model outperforms the state-of-the-art models in generating diverse distractors.

Cost awareness of radiological studies among doctors at Universitas Academic Hospital in Bloemfontein, South Africa.

BackgroundSouth Africa has high healthcare expenses. Improving cost-consciousness could decrease government expenditure on healthcare.ObjectivesTo determine cost awareness of radiological studies among doctors at a tertiary hospital. The objective was met by assessing the accuracy of cost estimation according to the level of training and speciality, whether participants had received prior education/training related to cost awareness and if they had a desire to learn more about the cost of radiological imaging.MethodA cross-sectional survey was conducted in six clinical departments at Universitas Academic Hospital using an anonymous questionnaire that determined doctors’ cost awareness of five radiological studies. Each radiological study was answered using six different cost ranges, with one correct option. Costs were based on the Department of Health’s 2019 Uniform Patients Fee Schedule (UPFS).ResultsIn total, 131 (67.2%) of 195 questionnaires distributed to registrars and consultants were returned. Overall, low accuracy of cost estimation was observed, with 45.2% of the participants choosing only incorrect options. No participant estimated all five costs correctly. Only the Internal Medicine clinicians demonstrated a significant difference between registrars and consultants for the number of correct answers (median 0 and 1, respectively) (p = 0.04). No significant differences were found between specialities stratified by registrars/consultants. Most participants (88.6%) would like to learn about imaging costs. Only 2.3% of the participants had received prior education/training related to cost awareness of radiological studies.ConclusionDoctors were consistently inaccurate in estimating the cost of radiological studies. Educating doctors about the cost of radiological imaging could have a positive effect on healthcare expenditure.

Task, usability, and error analyses of ambulance-based telemedicine for stroke care

Past research has established that telemedicine improves stroke care through decreased time to treatment and more accurate diagnoses. The goals of this study were to 1) study how clinicians complete stroke assessment using a telemedicine system integrated in ambulances, 2) determine potential errors and usability issues when using the system, and 3) develop recommendations to mitigate these issues. This study investigated use of a telemedicine platform to evaluate a stroke patient in an ambulance with a geographically distributed caregiving team comprised of a paramedic, nurse, and neurologist. It first determined the tasks involved based on 13 observations of a simulated stroke using 39 care providers. Based on these observational studies, a Hierarchical Task Analysis (HTA) was developed, and subsequently, a heuristic evaluation was conducted to determine the usability issues in the interface of the telemedicine system. This was followed by a Systematic Human Error Reduction and Prediction Approach (SHERPA) to determine the possibility of human error while providing care using the telemedicine work system. The results from the HTA included 6 primary subgoals categorizing the 97 tasks to complete the stroke evaluation. The heuristic evaluation found 123 unique violations to heuristics, with an average severity of 2.38. One hundred and thirty-one potential human errors were found with SHERPA, the two most common being miscommunication and selecting an incorrect option. Several recommendations are proposed, including improvement of labeling, consistent formatting, rigid or suggested formatting for data input, automation of task structure and camera movement, and audio/visual improvements to support communication.

Violation of the lexical norm in the modern newspaper language

This article assesses language as a key component in the development of media linguistics. Given that the main purpose of “language criticism” is to identify and correct shortcomings, it is im-portant to critically assess both incorrect and correct options when dealing with language pro-ducts. The paper puts special emphasis on the language of the media which needs to be normative. The author explores the development of newspaper language in terms of lexical norm vio la tion in the 1920s and 1930s. This article discusses equivalents to non-normative use of words in the daily all-Ukrainian newspaper The Day, one of the most popular publications in the ranking of national media. On the basis of various monolingual dictionaries of Standard Ukrainian and bilingual Russian-Ukrainian dictionaries, the author analyzes the use of tokens/lexemes napruha ‘voltage’, vidbyvaty‘to reflect’, pizniše ‘later’ in newspaper publications. Thus, the word napruha ‘voltage’ should be used only as a technical, physical term; in other cases it is napružennia ‘tension’. When it comes to expression, reproduction, an embodiment in images, it is correct to use vidobražaty ‘to reflect, re-mind of’, viddzerkaluvaty ‘to mirror’ but not vidbyvaty ‘to reflect’. As for pizniše ‘later’, it indicates a period after the time mentioned. It differs in its lexical meaning from zhodom ‘afterwards’ (some time after something) and potim ‘later’ (happening after something, following in time or space, no clear indication of time). Keywords: non-normative use, lexeme, dictionaries, newspaper publications.

The Role of Order and Sequence of Options in Multiple-choice Questions for High-stakes Tests of English Language Proficiency

High-stakes and high-volume English language proficiency tests typically rely on multiple-choice questions (MCQs) to assess reading and listening skills. Due to the Covid-19 pandemic, more institutions are using MCQs via online assessment platforms, which facilitate shuffling the order of options within test items to minimize cheating. There is scant research on the role that order and sequence of options plays in MCQs, so this study examined the results of a paper-based, high-stakes English proficiency test administered in two versions. Each version had identical three-option MCQs but with different ordering of options. The test-takers were chosen to ensure a very similar profile of language ability and level for the groups who took the two versions. The findings indicate that one in four questions exhibited significantly different levels of difficulty and discrimination between the two versions. The study identifies order dominance and sequence priming as two factors that influence the outcomes of MCQs, both of which can accentuate or diminish the power of attraction of the correct and incorrect options. These factors should be carefully considered when designing MCQs in high-stakes language proficiency tests and shuffling of options in either paper-based or computer-based testing.

The Properties of Powers: Didactic Contract and Gender Gap

National and international large-scale assessments of mathematics show that, in most nations, males achieve better results than females and Italy is one of the countries with a greater gap. Many research studies in mathematics education have analysed this issue, using both quantitative and qualitative methods to understand the sources and characteristics of this gap. This study focuses on a specific Grade 10 task that requires algebraic manipulations of powers with the same base. Item-level analysis enables the study of gender differences on specific content, before using the lenses of mathematics education theories to interpret macro-phenomena emergeing from standardized assessment results. The quantitative analysis, carried out using the Rasch statistical model, highlights a gender gap in favour of males in this task and, furthermore, a variance in choices of incorrect options between males and females; the interviews conducted provide a key to understanding this phenomenon in terms of didactic contract.

The Advantages of Five-Option Multiple-Choice Items in Classroom Tests of Student Mastery

The effectiveness of multiple-choice (MC) items depends on the quality of the response options—particularly how well the incorrect options (“distractors”) attract students who have incomplete knowledge. It is often contended that test-writers are unable to devise more than two plausible distractors for most MC items, and that the effort needed to do so is not worthwhile in terms of the items’ psychometric qualities. To test these contentions, I analyzed students’ performance on 545 MC items across six science courses that I have taught over the past decade. Each MC item contained four distractors, and the dataset included more than 19,000 individual responses. All four distractors were deemed plausible in one-third of the items, and three distractors were plausible in another third. Each increase in plausible distractor led to an average of a 13% increase in item difficulty. Moreover, an increase in plausible distractors led to a significant increase in the discriminability of the items, with a leveling off by the fourth distractor. These results suggest that—at least for teachers writing tests to assess mastery of course content—it may be worthwhile to eschew recent skepticism and continue to attempt to write MC items with three or four distractors.

Reply to comment on Howard et al. (2019): 'Nothing to dance about: unclear evidence for symbolic representations and numerical competence in honeybees'.

Reply to comment on Howard et al. (2019): 'Nothing to dance about: unclear evidence for symbolic representations and numerical competence in honeybees'.

Creating Diagnostic Assessments

The goal of this paper is to propose a new method to generate multiple-choice items that can make creating quality assessments faster and more efficient, solving a practical issue that many instructors face. There are currently no systematic, efficient methods available to generate quality distractors (plausible but incorrect options), which are necessary for multiple-choice assessments that accurately assess students’ knowledge. We propose two methods to use technology to generate quality multiple-choice assessments: (1) manipulating the mathematical problem to emulate common student misconceptions or errors and (2) disguising options to protect the integrity of multiple-choice tests. By linking options to common student misconceptions and errors, instructors can use assessments as personalized diagnostic tools that can target and modify underlying misconceptions. Moreover, using technology to generate these quality distractors would allow for assessments to be developed efficiently, in terms of both time and resources. The method to disguise the options generated would have the added benefit of preventing students from working backwards from options to solution and thus would protect the integrity of the assessment.