Generation Of Multiple Choice Questions Research Articles

LLM-Generated Multiple Choice Practice Quizzes for Pre-Clinical Medical Students; Use and Validity

Multiple choice questions (MCQs) are frequently used in medical education, and practice questions are commonly requested by students. Automated generation of MCQs in board-exam format would potentially save significant time for faculty and generate a wider set of practice materials. The goals of this study were to explore the feasibility of using ChatGPT by OpenAI to generate USMLE/COMLEX style practice quiz items as study aids, and to analyze how and to what extent medical students would voluntarily use these quiz items to prepare for exams. Researchers gave second year medical students studying renal physiology access to a set of practice quizzes with chatGPT generated questions. The number and timing of click throughs to the quizzes, number of question attempts, and scores of completed quizzes were anonymously recorded by custom quiz software. Practice questions were accessed 15,245 times by a class of 399 students. The majority of quiz click throughs occurred in the time immediately preceding exams. To analyze the quality of exam items, they were evaluated by independent experts for quality and adherence to USMLE/COMLEX guidelines. These results demonstrate the feasibility of LLM-generated practice questions in medical education when supervised by a human faculty. Supported By a Grant from the New York Institute of Technology Emerging Technology Group. This is the full abstract presented at the American Physiology Summit 2024 meeting and is only available in HTML format. There are no additional versions or additional content available for this abstract. Physiology was not involved in the peer review process.

The unified difficulty ranking mechanism for automatic multiple choice question generation in digital storytelling domain

The unified difficulty ranking mechanism for automatic multiple choice question generation in digital storytelling domain

Optimised NLP Model for MCQ Generation through Advanced Batching and Tokenization

Abstract: This research presents a pioneering methodology forenhancing Natural Language Processing (NLP) models through optimized Word Sense Disambiguation (WSD) and Multiple- Choice Question (MCQ) generation. By employing innovative strategies in batching and tokenization, this study revolutionizes the efficiency and accuracy of NLP tasks. This approach entails meticulous optimization of tokenization processes and concurrent batch operations, resulting in substantial computational efficien- cies without compromising the precision of WSD and MCQ generation. The proposed framework sets a new standard in NLP, offering robust enhancements in computational efficacy andlanguage comprehension tasks

Generation of Multiple-Choice Questions From Textbook Contents of School-Level Subjects

Multiple-choice question (MCQ) plays a significant role in educational assessment. Automatic MCQ generation has been an active research area for years, and many systems have been developed for MCQ generation. Still, we could not find any system that generates accurate MCQs from school-level textbook contents that are useful in real examinations. This observation motivated us to develop a system that generates MCQs to assess the student's recall of factual information. Also, the available systems are often domain, subject, or application-specific in nature. Although the MCQ generation task demands a specific setup, we expect a level of generalization can be achieved. In this development, we also focus on this issue. We propose a pipeline for automatic generation of MCQs from textbooks of middle-school level subjects, and the pipeline is partially subject-independent. The proposed pipeline comprises four core modules: preprocessing, sentence selection, key selection, and distractor generation. Several techniques have been employed to implement individual modules. These include sentence simplification, syntactic and semantic processing of the sentences, entity recognition, semantic relationship extraction among entities, WordNet, neural word embedding, neural sentence embedding, and computation of intersentence similarity. The system is evaluated using the National Council of Educational Research and Training (NCERT), India, textbooks for three subjects. The quality of system-generated questions is assessed by human experts using various system-level and individual module-level metrics. The experimental results demonstrate that the proposed system is capable of generating quality questions that could be useful in a real examination.

English grammar multiple-choice question generation using Text-to-Text Transfer Transformer

English grammar multiple-choice questions (MCQs) can be automatically generated to reduce preparation time. Previous studies have focused on semiautomated methods based on the transformation of human-made sentences/articles into MCQs, owing to which the number of generated questions is dependent on the size of a given text corpus. This study proposes an artificial intelligence-assisted MCQ generation system that increases the number of generable questions using controllable text generation techniques. In this system, the questions for MCQs are generated using a text generation model trained using the Text-to-Text Transfer Transformer (T5) architecture, a powerful deep learning model for performing text generation tasks, with a keyword and a part-of-speech (POS) template as the input for content and grammar topic control. For the text-to-MCQ transformation process, answer-to-MCQ and distractor-selection strategies are proposed for 10 grammar topics using rule-based algorithms. The quality of the generated MCQs is evaluated by human experts. The acceptance rate of the questions generated using the proposed system is 86%. The controllability of content and grammar topics are 96.86% and 98.57%, respectively. The findings of this study show that the T5 model achieves good performance in terms of controlling the POS structure in a keyword-to-text generation task. Moreover, the good acceptance rate indicates that artificial intelligence has the potential to help teachers speed up the process of selecting examination questions. We also discuss extending the proposed system to other grammar topics and the limitations of the proposed system.

Automatic computer science domain multiple-choice questions generation based on informative sentences.

Students require continuous feedback for effective learning. Multiple choice questions (MCQs) are extensively used among various assessment methods to provide such feedback. However, manual MCQ generation is a tedious task that requires significant effort, time, and domain knowledge. Therefore, a system must be present that can automatically generate MCQs from the given text. The automatic generation of MCQs can be carried out by following three sequential steps: extracting informative sentences from the textual data, identifying the key, and determining distractors. The dataset comprising of various topics from the 9th and 11th-grade computer science course books are used in this work. Moreover, TF-IDF, Jaccard similarity, quality phrase mining, K-means, and bidirectional encoder representation from transformers techniques are utilized for automatic MCQs generation. Domain experts validated the generated MCQs with 83%, 77%, and 80% accuracy, key generation, and distractor generation, respectively. The overall MCQ generation achieved 80% accuracy through this system by the experts. Finally, a desktop app was developed that takes the contents in textual form as input, processes it at the backend, and visualizes the generated MCQs on the interface. The presented solution may help teachers, students, and other stakeholders with automatic MCQ generation.

End-to-End generation of Multiple-Choice questions using Text-to-Text transfer Transformer models

The increasing worldwide adoption of e-learning tools and widespread increase of online education has brought multiple challenges, including the ability of generating assessments at the scale and speed demanded by this environment. In this sense, recent advances in language models and architectures like the Transformer, provide opportunities to explore how to assist educators in these tasks. This study focuses on using neural language models for the generation of questionnaires composed of multiple-choice questions, based on English Wikipedia articles as input. The problem is addressed using three dimensions: Question Generation (QG), Question Answering (QA), and Distractor Generation (DG). A processing pipeline based on pre-trained T5 language models is designed and a REST API is implemented for its use. The DG task is defined using a Text-To-Text format and a T5 model is fine-tuned on the DG-RACE dataset, showing an improvement to ROUGE-L metric compared to the reference for the dataset. A discussion about the lack of an adequate metric for DG is presented and the cosine similarity using word embeddings is considered as a complement. Questionnaires are evaluated by human experts reporting that questions and options are generally well formed, however, they are more oriented to measuring retention than comprehension.

Exploring (Collaborative) Generation and Exploitation of Multiple Choice Questions: Likes as Quality Proxy Metric

Multiple Choice Questions (MCQs) are an established medium of formal educational contexts. The collaborative generation of MCQs by students follows the perspectives of constructionist and situated learning and is an activity that fosters learning processes. The MCQs generated are—besides the learning processes—further outcomes of collaborative generation processes. Quality MCQs are a valuable resource, so that collaboratively generated quality MCQs might also be exploited in further educational scenarios. However, the quality MCQs first need to be identified from the corpus of all generated MCQs. This article investigates whether Likes distributed by students when answering MCQs are viable as a metric for identifying quality MCQs. Additionally, this study explores whether the process of collaboratively generating MCQs and using the quality MCQs generated in commercial quiz apps is achievable without additional extrinsic motivators. Accordingly, this article describes the results of a two-stage field study. The first stage investigates whether quality MCQs may be identified through collaborative inputs. For this purpose, the Reading Game (RG), a gamified, web-based software aiming at collaborative MCQ generation, is employed as a semester-accompanying learning activity in a bachelor course in Urban Water Management. The reliability of a proxy metric for quality calculated from the ratio of Likes received and appearances in quizzes is compared to the quality estimations of domain experts for selected MCQs. The selection comprised the ten best and the ten worst rated MCQs. Each of the MCQs is rated regarding five dimensions. The results support the assumption that the RG-given quality metric allows identification of well-designed MCQs. In the second stage, MCQs created by RG are provided in a commercial quiz app (QuizUp) in a voluntary educational scenario. Despite the prevailing pressure to learn, neither the motivational effects of RG nor of the app are found in this study to be sufficient for encouraging students to voluntarily use them on a regular basis. Besides confirming that quality MCQs may be generated by collaborative software, it is to be stated that in the collaborative generation of MCQs, Likes may serve as a proxy metric for the quality of the MCQs generated.

New multi-agent approach for generating feedbacks base on Multiple Choice Questions

Most intelligent tutorial systems promote the learning and resolution of exercises based on feedback in the form of advice, remarks, explanation…To always remain in self-assessment, our approach offers a new type of feedback in the form of multiple-choice questions applied in the field of algorithms (language c, java ...) dedicated for beginners in programming. The approach is based on the multi-agent model to have interaction between learners without the help of the teacher. There are three types of agents in our work, there is the learner agent who represents the learner, there is also the feedback agent who sends the MCQs to the learners who made mistakes and finally, the controller agent feeds the base of multiple-choice questions based on feedback from learners. The controlling agent compares the instructions of the learners with the instructions of the correct model based on the AST abstract syntax tree and detects errors which will be proposed as erroneous items (distractors) for the learners. We can say that this type of feedback is not direct like other work (advice, comments, explanation ...) but we always remain in the evaluation: MCQ, exercise ... As perspectives, we will focus mainly on the classification of LO learning objects in the form of an ontology to facilitate the use of data and the generation of multiple-choice questions at different levels. And also, we aim to develop a suitable platform which allows to define the agents and the messages to be exchanged between them to set up our system.

Multiple-choice question generation with auto-generated distractors for computer-assisted educational assessment

Multiple-choice question generation with auto-generated distractors for computer-assisted educational assessment

Affix-based Distractor Generation for Tamil Multiple Choice Questions using Neural Word Embedding

Assessment plays an important role in learning and Multiple Choice Questions (MCQs) are quite popular in large-scale evaluations. Technology enabled learning necessitates a smart assessment. Therefore, automatic MCQ generation became increasingly popular in the last two decades. Despite a large amount of research effort, system generated MCQs are not useful in real educational applications. This is because of the inability to produce the diverse and human alike distractors. Distractors are the wrong choices given along with the correct answer (key) to confuse the examinee. Especially, in educational domain (grammar learning) the MCQs deal with affix-based or morphologically transformed distractors. In this paper, we present a method for automatic generation of affix-based distractors for fill-in-the-blanks for learning Tamil Vocabulary. Affix-based distractor generation relies on certain regularities manifest in high dimensional spaces. We investigate the quality of distractors generated by a number of criteria, including Part-Of-Speech, difficulty level, spelling, word co-occurrence, semantic similarity and affixation. We evaluated our proposed method in grammar based Multiple Choice Questions (MCQs) dataset. The result shows that affix-based distractors, yield significantly more plausible outcomes in certain grammar based questions.

An Automated Multiple-Choice Question Generation using Natural Language Processing Techniques

Automatic multiple-choice question generation (MCQG) is a useful yet challenging task in Natural Language Processing (NLP). It is the task of automatic generation of correct and relevant questions from textual data. Despite its usefulness, manually creating sizeable, meaningful and relevant questions is a time-consuming and challenging task for teachers. In this paper, we present an NLP-based system for automatic MCQG for Computer-Based Testing Examination (CBTE).We used NLP technique to extract keywords that are important words in a given lesson material. To validate that the system is not perverse, five lesson materials were used to check the effectiveness and efficiency of the system. The manually extracted keywords by the teacher were compared to the auto-generated keywords and the result shows that the system was capable of extracting keywords from lesson materials in setting examinable questions. This outcome is presented in a user-friendly interface for easy accessibility.

Automatic Multiple Choice Question Generation From Text: A Survey

Automatic multiple choice question (MCQ) generation from a text is a popular research area. MCQs are widely accepted for large-scale assessment in various domains and applications. However, manual generation of MCQs is expensive and time-consuming. Therefore, researchers have been attracted toward automatic MCQ generation since the late 90's. Since then, many systems have been developed for MCQ generation. We perform a systematic review of those systems. This paper presents our findings on the review. We outline a generic workflow for an automatic MCQ generation system. The workflow consists of six phases. For each of these phases, we find and discuss the list of techniques adopted in the literature. We also study the evaluation techniques for assessing the quality of the system generated MCQs. Finally, we identify the areas where the current research focus should be directed toward enriching the literature.

Ontology-Based Generation of Medical, Multi-term MCQs

Designing good multiple choice questions (MCQs) for education and assessment is time consuming and error-prone. An abundance of structured and semi-structured data has led to the development of automatic MCQ generation methods. Recently, ontologies have emerged as powerful tools to enable the automatic generation of MCQs. However, current question generation approaches focus on knowledge recall questions. In addition, questions that have so far been generated are, compared to manually created ones, simple and cover only a small subset of the required question complexity space in the education and assessment domain. In this paper, we focus on addressing the limitations of previous approaches by generating questions with complex stems that are suitable for scenarios beyond mere knowledge recall. We present a novel ontology-based approach that exploits classes and existential restrictions to generate case-based questions. Our contribution lies in: (1) the specification of procedure for generating case-based questions which involve (a) assembling complex stems, (b) selecting suitable options, and (c) providing explanations for option correctness/incorrectness, (2) an implementation of the procedure using a medical ontology and (3) and evaluation of our generation technique to test question quality and their suitability in practise. We implement our approach as an application for a medical education scenario on top of a large knowledge base in the medical domain. We generate more than 3 million questions for four physician specialities and evaluate our approach in a user study with 15 medical experts. We find that using a stratified random sample of 435 questions out of which 316 were rated by two experts, 129 (30%) are considered appropriate to be used in exams by both experts and a further 216 (50%) by at least one expert.

A hybrid approach for automatic generation of named entity distractors for multiple choice questions

Assessment plays an important role in learning and Multiple Choice Questions (MCQs) are quite popular in large-scale evaluations. Technology-enabled learning necessitates a smart assessment. Therefore, automatic MCQ generation became increasingly popular in the last two decades. Despite a large amount of research effort, system generated MCQs are not useful in real educational applications. This is because of the inability to produce diverse and human-alike distractors. Distractors are the wrong choices given along with the correct answer (key) to befuddle the examinee. In several domains, the MCQs deal with names or named entities. However, existing literature is not adequate in generating quality named entity distractors. In this paper, we present a method for automatic generation of named entity distractors. The technique uses a combination of statistical and semantic similarities. To compute the statistical similarity, a set of class-specific attributes are defined are their values are extracted from the web. Semantic similarity is computed using a predicate-argument extraction based method. The proposed technique is tested in cricket domain because of the availability of a large number of web resources and MCQs for dataset preparation. An evaluation strategy is proposed along with a set of metrics. A set of human evaluators performed the evaluation and they found that the average closeness value of the distractors as 2.3. This value indicates that 2.3 out of 3 system-generated distractors are as good as human-generated distractors. Two good distractors make an MCQ usable in real assessment. So, the proposed technique is capable of generating high-quality distractors.

KHistory: A System for Automatic Generation of Multiple Choice Questions on the History of Korea

KHistory: A System for Automatic Generation of Multiple Choice Questions on the History of Korea

Automatic generation of multiple choice questions for e-assessment

It is important for students to expertise in their field of study, because there is an agile change in all the domains. Even though resources are available to learn, proper assessment helps them to improve upon their knowledge. In this paper, an automatic generation of multiple choice questions on any user-defined domain is proposed. It first extracts text relevant to the given domain from the web and summarises using fireflies-based preference learning. The sentences in the summary are transformed into stem for the MCQs. The distractors are generated using similarity metrics such as hypernyms and hyponyms. The system also generates analogy questions to test the verbal ability of the students.

Automatic generation of multiple choice questions for e-assessment

Automatic generation of multiple choice questions for e-assessment

Using Automatic Item Generation to Improve the Quality of MCQ Distractors

ABSTRACTConstruct: Automatic item generation (AIG) is an alternative method for producing large numbers of test items that integrate cognitive modeling with computer technology to systematically generate multiple-choice questions (MCQs). The purpose of our study is to describe and validate a method of generating plausible but incorrect distractors. Initial applications of AIG demonstrated its effectiveness in producing test items. However, expert review of the initial items identified a key limitation where the generation of implausible incorrect options, or distractors, might limit the applicability of items in real testing situations. Background: Medical educators require development of test items in large quantities to facilitate the continual assessment of student knowledge. Traditional item development processes are time-consuming and resource intensive. Studies have validated the quality of generated items through content expert review. However, no study has yet documented how generated items perform in a test administration. Moreover, no study has yet to validate AIG through student responses to generated test items. Approach: To validate our refined AIG method in generating plausible distractors, we collected psychometric evidence from a field test of the generated test items. A three-step process was used to generate test items in the area of jaundice. At least 455 Canadian and international medical graduates responded to each of the 13 generated items embedded in a high-stake exam administration. Item difficulty, discrimination, and index of discrimination estimates were calculated for the correct option as well as each distractor. Results: Item analysis results for the correct options suggest that the generated items measured candidate performances across a range of ability levels while providing a consistent level of discrimination for each item. Results for the distractors reveal that the generated items differentiated the low- from the high-performing candidates. Conclusions: Previous research on AIG highlighted how this item development method can be used to produce high-quality stems and correct options for MCQ exams. The purpose of the current study was to describe, illustrate, and evaluate a method for modeling plausible but incorrect options. Evidence provided in this study demonstrates that AIG can produce psychometrically sound test items. More important, by adapting the distractors to match the unique features presented in the stem and correct option, the generation of MCQs using automated procedure has the potential to produce plausible distractors and yield large numbers of high-quality items for medical education.

Ontology-Based Multiple Choice Question Generation

Multiple choice questions (MCQs) are considered highly useful (being easy to take or mark) but quite difficult to create and large numbers are needed to form valid exams and associated practice materials. The idea of re-using an existing ontology to generate MCQs almost suggests itself and has been explored in various projects. In this project, we are applying suitable educational theory regarding assessments and related methods for their evaluation to ontology-based MCQ generation. In particular, we investigate whether we can measure the similarity of the concepts in an ontology with sufficient reliability so that this measure can be used to control the difficulty of the MCQs generated. In this report, we provide an overview of the background to this research, and describe the main steps taken and insights gained.