Question Generation Research Articles

Examining the Validity and Reliability of ChatGPT 3.5-Generated Reading Comprehension Questions for Academic Texts

This research examines the capacity of ChatGPT 3.5 in generating reading comprehension questions for academic texts, with a focus on their alignment with higher-order cognitive skills as per Bloom’s Taxonomy. A paper-based test comprising 30 multiple-choice questions was constructed using ChatGPT 3.5, based on three selected TOEFL ITP reading comprehension passages. The study employed a mixed-methods approach, integrating qualitative content analysis to assess the cognitive level of each question and quantitative methods to analyze student responses. Data collection involved administering the AI-generated questions to students and scoring their responses. Analysis techniques included Pearson correlation coefficients to determine validity and reliability analysis using Cronbach's Alpha to measure internal consistency. The findings revealed that ChatGPT 3.5 is capable of producing questions that cover a range of cognitive levels, from analysis to creation, however only 10 out of 30 questions met the validity criteria, indicating a need for improvement in the AI's question generation process. The reliability of these questions was moderate, suggesting a reasonable level of internal consistency. The study concludes that while AI-generated questions show promise in educational assessments, ongoing improvement of AI models is necessary to enhance their effectiveness. The implications of this research are significant for the future integration of AI in educational settings, indicating a potential role for AI in developing meaningful assessment tools. The study recommends future research to explore various question types and incorporate student feedback to optimize the effectiveness of AI in education.

AN OVERVIEW OF EXISTING METHODS FOR AUTOMATIC GENERATION OF TEST TASKS IN NATURAL LANGUAGE

Recently, in the field of education, much attention has been paid to the use of multiple choice questions as a tool for assessing knowledge. The development of test tasks requires a lot of time and is highly labor intensive. It is difficult to perform such a task manually, so many researchers offer various ways and approaches to automate the creation of test tasks in natural language. In this paper, we present an overview of scientific achievements in the field of automatic question generation, which examines the classification of question generation systems by dividing them into five groups: machine learning-based methods, neural network-based, tree-based, rule-based or template-based and hybrid methods.

Large Language Models in Medical Education: Comparing ChatGPT- to Human-Generated Exam Questions.

Creating medical exam questions is time consuming, but well-written questions can be used for test-enhanced learning, which has been shown to have a positive effect on student learning. The automated generation of high-quality questions using large language models (LLMs), such as ChatGPT, would therefore be desirable. However, there are no current studies that compare students' performance on LLM-generated questions to questions developed by humans. The authors compared student performance on questions generated by ChatGPT (LLM questions) with questions created by medical educators (human questions). Two sets of 25 multiple-choice questions (MCQs) were created, each with 5 answer options, 1 of which was correct. The first set of questions was written by an experienced medical educator, and the second set was created by ChatGPT 3.5 after the authors identified learning objectives and extracted some specifications from the human questions. Students answered all questions in random order in a formative paper-and-pencil test that was offered leading up to the final summative neurophysiology exam (summer 2023). For each question, students also indicated whether they thought it had been written by a human or ChatGPT. The final data set consisted of 161 participants and 46 MCQs (25 human and 21 LLM questions). There was no statistically significant difference in item difficulty between the 2 question sets, but discriminatory power was statistically significantly higher in human than LLM questions (mean = .36, standard deviation [SD] = .09 vs mean = .24, SD = .14; P = .001). On average, students identified 57% of question sources (human or LLM) correctly. Future research should replicate the study procedure in other contexts (e.g., other medical subjects, semesters, countries, and languages). In addition, the question of whether LLMs are suitable for generating different question types, such as key feature questions, should be investigated.

Generation of Medical Case-Based Multiple-Choice Questions

This narrative review is a detailed look at how we make multiple-choice questions (MCQs) based on medical cases in today’s medical teaching. Moving from old-style MCQs to ones that are more related to real clinical situations is really important. It helps in growing critical thinking and practical use, especially since MCQs are still the primary method for testing knowledge in medicine. We look at the history, design ideas, and both manual and computer-based methods that have helped create MCQs. Technologies like Artificial Intelligence (AI) and Natural Language Processing (NLP) are receiving a lot of focus for their ability to automate the creation of question. We also talk about the challenges of using real patient cases, like the need for exact clinical information, reducing unclear information, and thinking about ethical issues. We also investigate the measures of validity and reliability that are crucial to maintaining the honesty of case-based MCQs. Finally, we look ahead, speculating on where medical education is headed as new technologies are incorporated and the value of case-based evaluations continues to rise.

Instrumental variables for implementation science: exploring context-dependent causal pathways between implementation strategies and evidence-based interventions

BackgroundThe impact of both implementation strategies (IS) and evidence-based interventions (EBI) can vary across contexts, and a better understanding of how and why this occurs presents fundamental but challenging questions that implementation science as a field will need to grapple with. We use causal epidemiologic methods to explore the mechanisms of why sharp distinctions between implementation strategies (IS) and efficacy of an evidence-based intervention (EBI) may fail to recognize that the effect of an EBI can be deeply intertwined and dependent on the context of the IS leading to its uptake.MethodsWe explore the use of instrumental variable (IV) analyses as a critical tool for implementation science methods to isolate three relevant quantities within the same intervention context when exposure to an implementation strategy is random: (1) the effect of an IS on implementation outcomes (e.g., uptake), (2) effect of EBI uptake on patient outcomes, and (3) overall effectiveness of the IS (i.e., ~ implementation*efficacy). We discuss the mechanisms by which an implementation strategy can alter the context, and therefore effect, of an EBI using the underlying IV assumptions. We illustrate these concepts using examples of the implementation of new ART initiation guidelines in Zambia and community-based masking programs in Bangladesh.ResultsCausal questions relevant to implementation science are answered at each stage of an IV analysis. The first stage assesses the effect of the IS (e.g., new guidelines) on EBI uptake (e.g., same-day treatment initiation). The second stage leverages the IS as an IV to estimate the complier average causal effect (CACE) of the EBI on patient outcomes (e.g., effect of same-day treatment initiation on viral suppression). The underlying assumptions of CACE formalize that the causal effect of EBI may differ in the context of a different IS because (1) the mechanisms by which individuals uptake an intervention may differ and (2) the subgroup of individuals who take up an EBI may differ. IV methods thus provide a conceptual framework for how IS and EBIs are linked and that the IS itself needs to be considered a critical contextual determinant. Moreover, it also provides rigorous methodologic tools to isolate the effect of an IS, EBI, and combined effect of the IS and EBI.DiscussionLeveraging IV methods when exposure to an implementation strategy is random helps to conceptualize the context-dependent nature of implementation strategies, EBIs, and patient outcomes. IV methods formalize that the causal effect of an EBI may be specific to the context of the implementation strategy used to promote uptake. This integration of implementation science concepts and theory with rigorous causal epidemiologic methods yields novel insights and provides important tools for exploring the next generation of questions related to mechanisms and context in implementation science.

HACAN: a hierarchical answer-aware and context-aware network for question generation

HACAN: a hierarchical answer-aware and context-aware network for question generation

Artificial Intelligence Empowers Gamification: Optimizing Student Engagement and Learning Outcomes in E-learning and MOOCs

In this era of Artificial Intelligence (AI) growth, characterized by advances in the Large Language Models (LLMs) used by ChatGPT and Bard, this study examines the effects of gamification and Automatic Question Generation (AQG) on student engagement and learning outcomes in the context of a Massive Open Online Course (MOOC). AQG, implemented via a Moodle plugin, transforms conventional assessments into an interactive, gamified experience, leveraging the “test effect” to improve learning outcomes. Research with 100 fifth-graders in a primary and secondary school shows that gamified assessments significantly boost student motivation and learning outcomes compared with traditional methods. The custom Moodle plugin facilitates the AQG process, generating contextually relevant and grammatically correct Multiple-Choice Questions (MCQs) from course content. The result is a dynamic, personalized assessment experience aimed at optimizing student retention. This paper concludes by discussing the implications of the study for educators and highlighting potential directions for future research.

Closing the Gap: Automated Distractor Generation in Japanese Language Testing

Recent advances in natural language processing have increased interest in automatic question generation, particularly in education (e.g., math, biology, law, medicine, and languages) due to its efficiency in assessing comprehension. Specifically, multiple-choice questions have become popular, especially in standardized language proficiency tests. However, manually creating high-quality tests is time-consuming and challenging. Distractor generation, a critical aspect of multiple-choice question creation, is often overlooked, yet it plays a crucial role in test quality. Generating appropriate distractors requires ensuring they are incorrect but related to the correct answer (semantically or contextually), are grammatically correct, and of similar length to the target word. While various languages have seen research in automatic distractor generation, Japanese has received limited attention. This paper addresses this gap by automatically generating cloze tests, including distractors, for Japanese language proficiency tests, evaluating the generated questions’ quality, difficulty, and preferred distractor types, and comparing them to human-made questions through automatic and manual evaluations.

Evidence-Based Learning Strategies in Medicine Using AI.

Large language models (LLMs), like ChatGPT, are transforming the landscape of medical education. They offer a vast range of applications, such as tutoring (personalized learning), patient simulation, generation of examination questions, and streamlined access to information. The rapid advancement of medical knowledge and the need for personalized learning underscore the relevance and timeliness of exploring innovative strategies for integrating artificial intelligence (AI) into medical education. In this paper, we propose coupling evidence-based learning strategies, such as active recall and memory cues, with AI to optimize learning. These strategies include the generation of tests, mnemonics, and visual cues.

Using Syntax and Shallow Semantic Analysis for Vietnamese Question Generation

Using Syntax and Shallow Semantic Analysis for Vietnamese Question Generation

Few-shot is enough: exploring ChatGPT prompt engineering method for automatic question generation in english education

Few-shot is enough: exploring ChatGPT prompt engineering method for automatic question generation in english education

A Framework for Assessing the Complexity of Auto Generated Questions from Ontologies

Automatic difficulty calibration (ADC) is the application of computational techniques to estimate the difficulty levels of assessment questions before administering them. Compared to traditional difficulty calibration approaches, ADC eliminates the need for pretesting, and minimises the time and efforts typically involved in manual calibration. In recent years, Ontology-based Automatic Question Generation (OAQG) has emerged as a powerful tool to generate assessment questions effortlessly and in massive numbers with minimal human intervention. Despite these benefits, not being able to control the characteristics of generated questions hinders their suitability to be used in pedagogical settings. However, much of the research up to now has tended to focus on the quantity of the questions rather than their quality. Indeed, most current automatic question generators produce simple questions which consist of a few facts, and simply test the recall of knowledge. Furthermore, the majority of existing frameworks are mostly technical and are not supported with a strong theoretical underpinning. In this paper, we propose a novel framework to assess the complexity of ontology-based, automatically generated questions. We discuss various aspects that are involved in determining the complexity of assessment questions, and attempt to quantify important characteristics of question complexity through the use of novel ontological metrics. To further support the plausibility of our computational framework, we shed light on its consistency with theories from education and cognitive psychology. This will provide us with a solid theoretical foundation which ensures that questions are generated according to principled methods that are grounded in theories of learning and cognition. The proposed approach is agnostic to different domains and independent of the question format, therefore, is highly general and applicable to a variety of contexts.

Context-Aware Auto-Encoded Graph Neural Model for Dynamic Question Generation using NLP

Question generation is an important task in natural language processing that involves generating questions from a given text. This paper proposes a novel approach for dynamic question generation using a context-aware auto-encoded graph neural model. Our approach involves constructing a graph representation of the input text, where each node in the graph corresponds to a word or phrase in the text, and the edges represent the relationships between them. We then use an auto-encoder model to learn a compressed representation of the graph that captures the most important information in the input text. Finally, we use the compressed graph representation to generate questions by dynamically selecting nodes and edges based on their relevance to the context of the input text. We evaluate our approach on four benchmark datasets (SQuAD, Natural Questions, TriviaQA, and QuAC) and demonstrate that it outperforms existing state-of-the-art methods for dynamic question generation. In the experimentation, to evaluate the result four performance metrics are used i.e. BLEU, ROUGE, F1-Score, and Accuracy. The result of the proposed approach yields an accuracy of 92% on the SQuAD dataset, 89% with QuAC, and 84% with TriviaQA. while on the natural questions dataset, the model gives 79% accuracy. Our results suggest that the use of graph neural networks and auto-encoder models can significantly improve the accuracy and effectiveness of question generation in NLP. Further research in this area can lead to even more sophisticated models that can generate questions that are even more contextually relevant and natural-sounding.

Novelty and Utility: How the Arts May Advance Question Creation in Contemporary Research

Abstract This paper builds on research around novelty and utility to argue that the value of arts thinking should be applied in the generation of scientific questions. Arts thinking is often playful, less goal oriented, and can lead to new modes of questioning. Scientific thinking often solves an existing question, serves a purpose in solving the question, and must be predictable. The “problem of the problem” is that asking creative questions is the linchpin of the quality of research across the sciences, just as the best of art “does things” that make us move and feel moved; yet we posit that it is useful to consider that what each teaches and celebrates typically tends more toward either utility or novelty as an entry point. A new theoretical basis is presented in identifying questions primarily based on novelty rather than utility, and a catalogue of methods proposed for creating questions to employ in education, practice, and project planning.

How will AI text generation and processing impact sustainability reporting? Critical analysis, a conceptual framework and avenues for future research

PurposeThe ability of generative artificial intelligence (AI) tools such as ChatGPT to produce convincing, human-like text has major implications for the future of corporate reporting, including sustainability reporting. As the importance of sustainability reporting continues to grow, this study aims to critically analyse the benefits and pitfalls of automated text generation and processing.Design/methodology/approachThis study develops a conceptual framework to delineate the field, assess the implications and form the basis for the generation of research questions. This study uses Alvesson and Deetz’s critical framework, considering insight (a review of literature and practice in the field), critique (consideration of the influences on the production and use of non-financial information and the implications for assurers of such information) and transformative redefinition (considering the implications of generative AI for sustainability reporting and proposing a research agenda).FindingsThis study highlights the implications of generative AI for sustainability accounting, reporting, assurance and report usage, including the risk of AI facilitating greenwashing, and the importance of more research on the use of AI for these matters.Practical implicationsThe paper highlights to stakeholders the implications of AI for all aspects of sustainability reporting, including accounting, reporting, assurance and usage of reports.Social implicationsThe implications of AI need to be understood in society, which this paper facilitates.Originality/valueThis study critically analyses the potential use of AI for sustainability reporting, construct a conceptual framework to delineate the field and develop a research agenda.

Automated System for Generating and Evaluating Tests

The importance of knowledge testing in the educational process and its role in identifying gaps in students’ knowledge and modernizing education are noted. The contextual links between the concept of quality and educational testing, the representation of knowledge about the subject area using the Martynov triad are considered. The main focus is on the semantic web, where concepts represent testable knowledge elements. Concepts are classified into different types depending on the information they contain, such as general textual concept, list/classification/hierarchy, program code, formulaic concept, causal concept, and others. The generation of test questions is considered, the QueTabAI system is described, which automatically generates questions based on the presented knowledge about the subject area. Methods for generating questions with their compilation on the basis of grammatical analysis of the text are given. The signs or criteria that are identified through testing are listed. Various aspects and methods of feature formation are studied, allowing to make a decision on the need to fill in the knowledge gaps and modernize the educational process.

Graph convolutional network for difficulty-controllable visual question generation

Graph convolutional network for difficulty-controllable visual question generation

A Cross-Disciplinary Examination of the Instructional Uses of ChatGPT in Higher Education

This paper examines the potential uses of ChatGPT in generating assessment tasks that can be used across different disciplines in higher education. To illustrate this, we provide examples from three courses in the disciplines of industrial engineering and applied linguistics: Project Analysis and Control, Manufacturing Processes, and Introduction to Linguistics. Our examination of ChatGPT focuses on how some of the common, but time-consuming, tasks can be generated by using ChatGPT as a supportive instructional tool. We observed in our analysis that ChatGPT demonstrates a high level of performance in generating assessment questions and tasks that are accurate and on-topic, and a level of creativity and flexibility in its question generation capabilities. However, it is important to note that ChatGPT is not designed to replace human expertise or judgment. It is crucial that instructors carefully evaluate the reliability and accuracy of the assessments or information generated by ChatGPT.

Visual question generation for explicit questioning purposes based on target objects

Visual question generation aims to focus on some target objects in an image to generate questions with certain questioning purposes. Existing studies mainly utilize an answer to extract the target object corresponding to the questioning purpose for questioning. However, answers fail to accurately and completely map to every target object, such as the objects corresponding to the answer are ambiguous or the answers are the relationship between multiple objects. To address this problem, we propose a content-controlled question generation model, which generates questions based on a given target object set specified from an image. Considering that the target objects have different contributions during the generation process, we design a recurrent generative architecture to explicitly control attention to different objects and their corresponding image information at each generative stage. Extensive experiments on the VQA v2.0 dataset and the Visual7w dataset show that the proposed model outperforms the state-of-the-art models and can controllably generate questions with specified content.

Automatic Skill-Oriented Question Generation and Recommendation for Intelligent Job Interviews

Job interviews are the most widely accepted method for companies to select suitable candidates, and a critical challenge is finding the right questions to ask job candidates. Moreover, there is a lack of integrated tools for automatically generating interview questions and recommending the right questions to interviewers. To this end, in this paper, we propose an intelligent system for assisting job interviews, namely, DuerQues. To build this system, we first investigate how to automatically generate skill-oriented interview questions in a scalable way by learning external knowledge from online knowledge-sharing communities. Along this line, we develop a novel distantly supervised skill entity recognition method to identify skill entities from large-scale search queries and web page titles with less need for human annotation. Additionally, we propose a neural generative model for generating skill-oriented interview questions. In particular, we introduce a data-driven solution to create high-quality training instances and design a learning algorithm to improve the performance of question generation. Furthermore, we exploit click-through data from query logs and design a recommender system for recommending suitable questions to interviewers. Specifically, we introduce a graph-enhanced algorithm to efficiently recommend suitable questions given a set of queried skills. Finally, extensive experiments on real-world datasets demonstrate the effectiveness of our DuerQues system in terms of the quality of generated skill-oriented questions and the performance of question recommendation.