Distractor Generation Research Articles

SExpSMA-based T5: Serial exponential-slime mould algorithm based T5 model for question answer and distractor generation

Generally, multiple choice questions are an effective and extensive form used in standard tests in order to evaluate the learner’s skills and knowledge. Nonetheless, the multiple-choice question composition particularly the distractor construction is quite difficult. The distracters are needed to be both plausible and inappropriate and adequate to mystify the learners who did not master the information. Thus, the distractor generation emergence is important that can help several standard tests in an extensive range of domain. In this research, question-answer generation system is developed with a distractor model by developing an optimized T5 model. At first, BERT tokenization is used to pre-process the passage/context and question, which are given as the input to train the approach. Then, the question and answer generation is performed by utilizing the T5 approach that is trained by proposed Serial Exponential-Slime Mould approach (SExpSMA). Exponential weighted moving average is extended to Serial Exponential weighted moving average and incorporated in Slime Mould Algorithm (SMA) to propose SExpSMA. In addition, the proposed SExpSMA-based T5 model is employed to generate distractors for the questions. Eventually, experimentation analysis exhibits that proposed SExpSMA-based T5 model achieves better outcomes regarding the metrics, like ROUGE, BLEU, and METEOR with the values of 0.919, 0.918, 0.488, respectively.

Distractor Generation Through Text-to-Text Transformer Models

Distractor Generation Through Text-to-Text Transformer Models

Closing the Gap: Automated Distractor Generation in Japanese Language Testing

Recent advances in natural language processing have increased interest in automatic question generation, particularly in education (e.g., math, biology, law, medicine, and languages) due to its efficiency in assessing comprehension. Specifically, multiple-choice questions have become popular, especially in standardized language proficiency tests. However, manually creating high-quality tests is time-consuming and challenging. Distractor generation, a critical aspect of multiple-choice question creation, is often overlooked, yet it plays a crucial role in test quality. Generating appropriate distractors requires ensuring they are incorrect but related to the correct answer (semantically or contextually), are grammatically correct, and of similar length to the target word. While various languages have seen research in automatic distractor generation, Japanese has received limited attention. This paper addresses this gap by automatically generating cloze tests, including distractors, for Japanese language proficiency tests, evaluating the generated questions’ quality, difficulty, and preferred distractor types, and comparing them to human-made questions through automatic and manual evaluations.

Distractor Generation for Lexical Questions Using Learner Corpus Data

Abstract Learner corpora with error annotation can serve as a source of data for automated question generation (QG) for language testing. In case of multiple choice gapfill lexical questions, this process involves two steps. The first step is to extract sentences with lexical corrections from the learner corpus. The second step, which is the focus of this paper, is to generate distractors for the retrieved questions. The presented approach (called DisSelector) is based on supervised learning on specially annotated learner corpus data. For each sentence a list of distractor candidates was retrieved. Then, each candidate was manually labelled as a plausible or implausible distractor. The derived set of examples was additionally filtered by a set of lexical and grammatical rules and then split into training and testing subsets in 4:1 ratio. Several classification models, including classical machine learning algorithms and gradient boosting implementations, were trained on the data. Word and sentence vectors from language models together with corpus word frequencies were used as input features for the classifiers. The highest F1-score (0.72) was attained by a XGBoost model. Various configurations of DisSelector showed improvements over the unsupervised baseline in both automatic and expert evaluation. DisSelector was integrated into an opensource language testing platform LangExBank as a microservice with a REST API.

Using Automatic Item Generation to Create Multiple-Choice Questions for Pharmacy Assessment

ObjectiveAutomatic item generation (AIG) is a new area of assessment research where a set of multiple-choice questions (MCQs) are created using models and computer technology. Although successfully demonstrated in medicine and dentistry, AIG has not been implemented in pharmacy. The objective was to implement AIG to create a set of MCQs appropriate for inclusion in a summative, high-stakes, pharmacy examination. MethodsA 3-step process, well evidenced in AIG research, was employed to create the pharmacy MCQs. The first step was developing a cognitive model based on content within the examination blueprint. Second, an item model was developed based on the cognitive model. A process of systematic distractor generation was also incorporated to optimize distractor plausibility. Third, we used computer technology to assemble a set of test items based on the cognitive and item models. A sample of generated items was assessed for quality against Gierl and Lai’s 8 guidelines of item quality. ResultsMore than 15,000 MCQs were generated to measure knowledge and skill of patient assessment and treatment of nausea and/or vomiting within the scope of clinical pharmacy. A sample of generated items satisfies the requirements of content-related validity and quality after substantive review. ConclusionThis research demonstrates the AIG process is a viable strategy for creating a test item bank to provide MCQs appropriate for inclusion in a pharmacy licensing examination.

Generation of Multiple-Choice Questions From Textbook Contents of School-Level Subjects

Multiple-choice question (MCQ) plays a significant role in educational assessment. Automatic MCQ generation has been an active research area for years, and many systems have been developed for MCQ generation. Still, we could not find any system that generates accurate MCQs from school-level textbook contents that are useful in real examinations. This observation motivated us to develop a system that generates MCQs to assess the student's recall of factual information. Also, the available systems are often domain, subject, or application-specific in nature. Although the MCQ generation task demands a specific setup, we expect a level of generalization can be achieved. In this development, we also focus on this issue. We propose a pipeline for automatic generation of MCQs from textbooks of middle-school level subjects, and the pipeline is partially subject-independent. The proposed pipeline comprises four core modules: preprocessing, sentence selection, key selection, and distractor generation. Several techniques have been employed to implement individual modules. These include sentence simplification, syntactic and semantic processing of the sentences, entity recognition, semantic relationship extraction among entities, WordNet, neural word embedding, neural sentence embedding, and computation of intersentence similarity. The system is evaluated using the National Council of Educational Research and Training (NCERT), India, textbooks for three subjects. The quality of system-generated questions is assessed by human experts using various system-level and individual module-level metrics. The experimental results demonstrate that the proposed system is capable of generating quality questions that could be useful in a real examination.

EduQG: A Multi-Format Multiple-Choice Dataset for the Educational Domain

Natural language processing technology has made significant progress in recent years, fuelled by increasingly powerful general language models. This has also inspired a sizeable body of work targeted specifically towards the educational domain, where the creation of questions (both for assessment and practice) is a laborious/expensive effort. Thus, automatic Question-Generation (QG) solutions have been proposed and studied. Yet, according to a recent survey of the educational QG community’s progress, a common baseline dataset unifying multiple domains and question forms (e.g., multiple choice vs. fill-the-gap), including readily available baseline models to compare against, is largely missing. This is the gap we aim to fill with this paper. In particular, we introduce a high-quality dataset in the educational domain, containing over 3,000 entries, comprising (i) multiple-choice questions, (ii) the corresponding answers (including distractors), and (iii) associated passages from the course material used as sources for the questions. Each question is phrased in two forms, normal and cloze (i.e., fill-the-gap), and correct answers are linked to source documents with sentence-level annotations. Thus, our versatile dataset can be used for both question and distractor generation, as well as to explore new challenges such as question format conversion. Furthermore, 903 questions are accompanied by their cognitive complexity level as per Bloom’s taxonomy. All questions have been generated by educational experts rather than crowd workers to ensure they are maintaining educational and learning standards. Our analysis and experiments suggest distinguishable differences between our dataset and commonly used ones for question generation for educational purposes. We believe this new dataset can serve as a valuable resource for research and evaluation in the educational domain. The dataset and baselines are made available to support further research in question generation for education (https://github.com/hadif ar/question-generation).

Automatic computer science domain multiple-choice questions generation based on informative sentences.

Students require continuous feedback for effective learning. Multiple choice questions (MCQs) are extensively used among various assessment methods to provide such feedback. However, manual MCQ generation is a tedious task that requires significant effort, time, and domain knowledge. Therefore, a system must be present that can automatically generate MCQs from the given text. The automatic generation of MCQs can be carried out by following three sequential steps: extracting informative sentences from the textual data, identifying the key, and determining distractors. The dataset comprising of various topics from the 9th and 11th-grade computer science course books are used in this work. Moreover, TF-IDF, Jaccard similarity, quality phrase mining, K-means, and bidirectional encoder representation from transformers techniques are utilized for automatic MCQs generation. Domain experts validated the generated MCQs with 83%, 77%, and 80% accuracy, key generation, and distractor generation, respectively. The overall MCQ generation achieved 80% accuracy through this system by the experts. Finally, a desktop app was developed that takes the contents in textual form as input, processes it at the backend, and visualizes the generated MCQs on the interface. The presented solution may help teachers, students, and other stakeholders with automatic MCQ generation.

End-to-End generation of Multiple-Choice questions using Text-to-Text transfer Transformer models

The increasing worldwide adoption of e-learning tools and widespread increase of online education has brought multiple challenges, including the ability of generating assessments at the scale and speed demanded by this environment. In this sense, recent advances in language models and architectures like the Transformer, provide opportunities to explore how to assist educators in these tasks. This study focuses on using neural language models for the generation of questionnaires composed of multiple-choice questions, based on English Wikipedia articles as input. The problem is addressed using three dimensions: Question Generation (QG), Question Answering (QA), and Distractor Generation (DG). A processing pipeline based on pre-trained T5 language models is designed and a REST API is implemented for its use. The DG task is defined using a Text-To-Text format and a T5 model is fine-tuned on the DG-RACE dataset, showing an improvement to ROUGE-L metric compared to the reference for the dataset. A discussion about the lack of an adequate metric for DG is presented and the cosine similarity using word embeddings is considered as a complement. Questionnaires are evaluated by human experts reporting that questions and options are generally well formed, however, they are more oriented to measuring retention than comprehension.

State-of-the-Art Approach to e-Learning with Cutting Edge NLP Transformers: Implementing Text Summarization, Question and Distractor Generation, Question Answering

Amid the worldwide wave of pandemic lockdowns, there has been a remarkable growth in E-learning. Online learning has become a challenge for students. It has become difficult for students to find the content they need. The mounting accessibility of textual content has necessitated comprehensive study in the areas of automatic text summarization and question generation. Multiple Choice Questions is very smooth for evaluations, and its assessment is implemented through computerized applications in order that results may be declared within some hours, and the evaluation system is 100% pure. The system proposes an interactive reading platform where the user can upload an E-Book and get textual summary and generates questions like MCQs, fill in the blanks and one word. The user can also evaluate the questions answered. The proposed system is an all-in-one interactive reading platform.

Diverse Distractor Generation for Constructing High-Quality Multiple Choice Questions

Distractor generation task aims to generate incorrect options (i.e., distractors) for multiple choice questions from an article.Existing methods for this task often utilize a standard encoder-decoder framework. However, these methods often tend to generate semantically similar distractors, since the same article representations are used to generate different distractors. Multiple generated distractors with similar semantics are considered equivalent. Because the correct answer is unique, students can eliminate these distractors even without reading the article. In this paper, we propose a multi-selector generation network (MSG-Net) that generates distractors with rich semantics based on different sentences in an article. MSG-Net adopts a multi-selector mechanism to select multiple different sentences in an article that are useful to generate diverse distractors. Specifically, a question-aware and answer-aware mechanism are introduced to assist in selecting useful key sentences, where each key sentence is coherent with the question and not equivalent to the answer. MSG-Net can generate diverse distractors based on each selected key sentence with different semantics. Extensive experiments on the RACE dataset and Cosmos QA dataset show that the proposed model outperforms the state-of-the-art models in generating diverse distractors.

An Evidence-Based Approach to Distractor Generation in Multiple-Choice Language Tests

The purpose of this project is to explore the feasibility of a new approach for producing evidence-based distractor sets. We use Common Wrong Answers (CWAs), and the associated performance data, generated by candidate responses to open gap-fill tasks, to produce distractor sets for multiple-choice gap-fill tasks based on the same texts. We then investigate whether these distractor sets are effective for use in language tests, in terms of empirical and qualitative review, and consider potential impacts on the production process for test material. This project explores a new and innovative method of content development, and raises the possibility of a new approach to item production that can semi-autogenerate test items in shorter periods of time without affecting quality or reliability. Although the approach is specific to one task type, it is hoped that further research will expand on the applications of the approach to deliver a version that may be operationalised for use across different task types in the development of language assessments.

Automatic Morpheme-based Distractors Generation for Fill-in-the-Blank Questions using Listwise Learning-To-Rank Method for Agglutinative Language

Automatic question generation facilitates the smart assessment for the evaluator to assess the student skills. Several methods were proposed to generate distractors for non-factoid cloze question using different similarity measures. This study presents a method for automatic generation of affix based distractor for Tamil fill-in-the-blank questions which are mainly used for learning Tamil grammar morphological details and vocabulary. In this study, affix based distractor generation is proposed as two step pipelined process: 1) Distractor candidate collection: This generation mainly relies on certain regularities manifest in high dimensional spaces which implicitly hybrids the orthographic and semantic features. 2) Distractor filtering: Filtering is trained as Learning-to-Rank models to persist the reliability in distractor generation. Feature based Listwise approaches (ListNet and ListMLE) were used which uses caserole relationship, subject-verb agreement, POS tag in addition to similarity measures. Experiments were done with annotated dataset (TamilMCQs) taken from 5th to 12th grade Tamil text books. Experimental results show that hybridization of spelling and semantic features highly improves the plausibility in distractor generation and then, ListMLE method improves the reliability in distractor generation compared to ListNet method. As a overall, our proposed pipelined process increases plausibility and reliability in distractor generation.

Knowledge-Driven Distractor Generation for Cloze-Style Multiple Choice Questions

In this paper, we propose a novel configurable framework to automatically generate distractive choices for open-domain cloze-style multiple-choice questions. The framework incorporates a general-purpose knowledge base to effectively create a small distractor candidate set, and a feature-rich learning-to-rank model to select distractors that are both plausible and reliable. Experimental results on a new dataset across four domains show that our framework yields distractors outperforming previous methods both by automatic and human evaluation. The dataset can also be used as a benchmark for distractor generation research in the future.

Affix-based Distractor Generation for Tamil Multiple Choice Questions using Neural Word Embedding

Assessment plays an important role in learning and Multiple Choice Questions (MCQs) are quite popular in large-scale evaluations. Technology enabled learning necessitates a smart assessment. Therefore, automatic MCQ generation became increasingly popular in the last two decades. Despite a large amount of research effort, system generated MCQs are not useful in real educational applications. This is because of the inability to produce the diverse and human alike distractors. Distractors are the wrong choices given along with the correct answer (key) to confuse the examinee. Especially, in educational domain (grammar learning) the MCQs deal with affix-based or morphologically transformed distractors. In this paper, we present a method for automatic generation of affix-based distractors for fill-in-the-blanks for learning Tamil Vocabulary. Affix-based distractor generation relies on certain regularities manifest in high dimensional spaces. We investigate the quality of distractors generated by a number of criteria, including Part-Of-Speech, difficulty level, spelling, word co-occurrence, semantic similarity and affixation. We evaluated our proposed method in grammar based Multiple Choice Questions (MCQs) dataset. The result shows that affix-based distractors, yield significantly more plausible outcomes in certain grammar based questions.

WITHDRAWN: Distractor filtering by context aware models in MCQs

Abstract Assessment of knowledge in the usage of affix based words is very much essential in the context of learning any language. Designing such questions focusing on the morphological aspects in the automated way is a challenging one. Especially, generating the distractors w.r.t. MCQs automatically concerned with affix words in Tamil Language is focused in this paper. In this study, we mainly focused on distractor filtering through context aware language models by two approaches. First one, list is filtered through masked word prediction from pre-trained BERT model. Next, filtering is formulated as sentence classification which implicitly checks the sentence completion in terms of morphemes. Classification model is fine-tuned from pre-trained multilingual model. Experimental results show that masked language model outperforms the classification model due to low dataset used in fine-tuning process. Compared to trigram filtering approach context aware language model optimizes reliability in affix based distractor generation.

Co-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension

In reading comprehension, generating sentence-level distractors is a significant task, which requires a deep understanding of the article and question. The traditional entity-centered methods can only generate word-level or phrase-level distractors. Although recently proposed neural-based methods like sequence-to-sequence (Seq2Seq) model show great potential in generating creative text, the previous neural methods for distractor generation ignore two important aspects. First, they didn't model the interactions between the article and question, making the generated distractors tend to be too general or not relevant to question context. Second, they didn't emphasize the relationship between the distractor and article, making the generated distractors not semantically relevant to the article and thus fail to form a set of meaningful options. To solve the first problem, we propose a co-attention enhanced hierarchical architecture to better capture the interactions between the article and question, thus guide the decoder to generate more coherent distractors. To alleviate the second problem, we add an additional semantic similarity loss to push the generated distractors more relevant to the article. Experimental results show that our model outperforms several strong baselines on automatic metrics, achieving state-of-the-art performance. Further human evaluation indicates that our generated distractors are more coherent and more educative compared with those distractors generated by baselines.

Automatic generation of fill‐in‐the‐blank question with corpus‐based distractors for e‐assessment to enhance learning

AbstractKnowledge acquisition is the prime objective of a learner from an educational system and evaluating the learner's knowledge is the eventual goal of an examination process. This paper introduces a system which is able to produce fill‐in‐the‐blank questions to test the knowledge of a learner that he or she has accumulated after reading a course material. The question generation task is subdivided into three modules: sentence selection, answer‐key identification, and question formation along with distractor generation. The sentence is selected using a coarse‐grain part‐of‐speech tagset. The answer‐key is extracted by identifying topic‐word in the sentence and question is formed by omitting this topic‐word from the sentence. This paper also highlights an efficient corpus‐based distractor generation technique to produce multiple‐choice fill‐in‐the‐blank test items.

Generating Distractors for Reading Comprehension Questions from Real Examinations

We investigate the task of distractor generation for multiple choice reading comprehension questions from examinations. In contrast to all previous works, we do not aim at preparing words or short phrases distractors, instead, we endeavor to generate longer and semantic-rich distractors which are closer to distractors in real reading comprehension from examinations. Taking a reading comprehension article, a pair of question and its correct option as input, our goal is to generate several distractors which are somehow related to the answer, consistent with the semantic context of the question and have some trace in the article. We propose a hierarchical encoderdecoder framework with static and dynamic attention mechanisms to tackle this task. Specifically, the dynamic attention can combine sentence-level and word-level attention varying at each recurrent time step to generate a more readable sequence. The static attention is to modulate the dynamic attention not to focus on question irrelevant sentences or sentences which contribute to the correct option. Our proposed framework outperforms several strong baselines on the first prepared distractor generation dataset of real reading comprehension questions. For human evaluation, compared with those distractors generated by baselines, our generated distractors are more functional to confuse the annotators.

Automatic Fill in the Blank Question with Distractor Generation Using NLP

International Journal of Computer Sciences and Engineering (A UGC Approved and indexed with DOI, ICI and Approved, DPI Digital Library) is one of the leading and growing open access, peer-reviewed, monthly, and scientific research journal for scientists, engineers, research scholars, and academicians, which gains a foothold in Asia and opens to the world, aims to publish original, theoretical and practical advances in Computer Science,Information Technology, Engineering (Software, Mechanical, Civil, Electronics & Electrical), and all interdisciplinary streams of Computing Sciences. It intends to disseminate original, scientific, theoretical or applied research in the field of Computer Sciences and allied fields. It provides a platform for publishing results and research with a strong empirical component. It aims to bridge the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research.