Reading Comprehension Datasets Research Articles

Combining permuted language model and adversarial training for Chinese machine reading comprehension

Due to the polysemy and complexity of the Chinese language, Chinese machine reading comprehension has always been a challenging task. To improve the semantic understanding and robustness of Chinese machine reading comprehension models, we propose a model that utilizes adversarial training algorithms and Permuted Language Model (PERT). Firstly, we employ the PERT pre-training model to embed paragraphs and questions into vector space to obtain corresponding sequential representations. Secondly, we use a multi-head self-attention mechanism to extract key textual information from the sequence and employ a Bi-GRU network to semantically fuse the output feature vectors, aiming to learn deep semantic representations in the text. Finally, we introduce perturbations into the model training process. We achieve this by utilizing adversarial training algorithms such as Fast Gradient Method (FGM) and Projected Gradient Descent (PGD). These algorithms generate adversarial samples to enhance the model’s robustness and stability when facing diverse inputs. We conducted comparative experiments on the publicly available Chinese reading comprehension datasets CMRC2018 and DRCD. The experimental results show that our proposed model has achieved significant improvements in both EM and F1-Score compared to the baseline model. To validate the model’s generalization and robustness, we utilized ChatGPT to construct a scientific dataset that includes a large number of domain-specific terms, sentences with mixed Chinese and English, and complex comprehension tasks. Our model also performed remarkably well on the self-built dataset. In conclusion, the proposed model not only effectively enhances the understanding of semantic information in Chinese text but also demonstrates a certain level of generalization capability.

Weight Averaging and re-adjustment ensemble for QRCD

Question Answering (QA) is a prominent task in the field of Natural Language Processing (NLP) with extensive applications. Recently, there has been a notable surge in research interest concerning the development of QA systems for the Holy Qur’an, an Islamic religious text. The Qur’an Reading Comprehension Dataset (QRCD) Malhas and Elsayed (2020) is a highly commendable effort in this respect. It stands as the first benchmark dataset specifically designed for a set of directly answerable questions from the Qur’an. Each question in the dataset is meticulously labeled with all potential answers sourced from the Holy Qur’an. From our perspective, the main challenge in QRCD stems from the limited volume of training data it offers. As a solution we propose an innovative approach to build a Deep Neural Network (DNN) ensemble, centered around Ara-Electra model (Antoun et al., 2021), that we called Weight Averaging and Re-adjustment (WAR) model. The model is constructed by computing running averages of all model states that evolve during a single training session and ensuring that model weights are readjusted prior to each training epoch, in order to hold it back from over fitting the training data. The scheme results in a single standalone model that exhibits the benefits of multi-model ensembles. It is distinguished from other ensembles proposed for QRCD that accumulates outputs from multiple expert models and employs classic techniques like hard voting or score averaging on output probabilities to build unified results. Each expert model costs individual training time and compute resources. The WAR model outperforms existing systems with improved generalization over unseen data. It achieves F1, partial Reciprocal Rank (pRR), and exact-match (EM) scores of 0.567, 0.60 and 0.29 respectively, exceeding best reported QRCD scores by 3%, 1.5% and 0.69% respectively. Notably, we are comparing our results with the top scores from different models, highlighting our model’s consistent performance across all three metrics.

Mitigating Idiom Inconsistency: A Multi-Semantic Contrastive Learning Method for Chinese Idiom Reading Comprehension

Chinese idioms pose a significant challenge for machine reading comprehension due to their metaphorical meanings often diverging from their literal counterparts, leading to metaphorical inconsistency. Furthermore, the same idiom can have different meanings in different contexts, resulting in contextual inconsistency. Although deep learning-based methods have achieved some success in idioms reading comprehension, existing approaches still struggle to accurately capture idiom representations due to metaphorical inconsistency and contextual inconsistency of idioms. To address these challenges, we propose a novel model, Multi-Semantic Contrastive Learning Method (MSCLM), which simultaneously addresses metaphorical inconsistency and contextual inconsistency of idioms. To mitigate metaphorical inconsistency, we propose a metaphor contrastive learning module based on the prompt method, bridging the semantic gap between literal and metaphorical meanings of idioms. To mitigate contextual inconsistency, we propose a multi-semantic cross-attention module to explore semantic features between different metaphors of the same idiom in various contexts. Our model has been compared with multiple current latest models (including GPT-3.5) on multiple Chinese idiom reading comprehension datasets, and the experimental results demonstrate that MSCLM outperforms state-of-the-art models.

Benchmarks for Pirá 2.0, a Reading Comprehension Dataset about the Ocean, the Brazilian Coast, and Climate Change

ABSTRACT Pirá is a reading comprehension dataset focused on the ocean, the Brazilian coast, and climate change, built from a collection of scientific abstracts and reports on these topics. This dataset represents a versatile language resource, particularly useful for testing the ability of current machine learning models to acquire expert scientific knowledge. Despite its potential, a detailed set of baselines has not yet been developed for Pirá. By creating these baselines, researchers can more easily utilize Pirá as a resource for testing machine learning models across a wide range of question answering tasks. In this paper, we define six benchmarks over the Pirá dataset, covering closed generative question answering, machine reading comprehension, information retrieval, open question answering, answer triggering, and multiple choice question answering. As part of this effort, we have also produced a curated version of the original dataset, where we fixed a number of grammar issues, repetitions, and other shortcomings. Furthermore, the dataset has been extended in several new directions, so as to face the aforementioned benchmarks: translation of supporting texts from English into Portuguese, classification labels for answerability, automatic paraphrases of questions and answers, and multiple choice candidates. The results described in this paper provide several points of reference for researchers interested in exploring the challenges provided by the Pirá dataset.

Combined Approach for Answer Identification with Small Sized Reading Comprehension Datasets

Combined Approach for Answer Identification with Small Sized Reading Comprehension Datasets

Verb-driven machine reading comprehension with dual-graph neural network

Logical reasoning of context is vital for reading comprehension, which requires to explore the logical relationship through sentence structure. However, previous methods of logical symbols and graph-based models do not make full explore the relationships among entities. In this paper, we present a verb-driven dual-graph network (VDGN) that utilizes core verbs of sentences to model the inter-sentence relationship by the ability of verbs to express linguistic context and the shortest dependency path to model the relationship between entities of intra-sentence. We construct a context graph and a query graph respectively through the above method. In order to predict the answer correctly, our framework fuses information from the context graph and the query graph applying a bi-directional attention mechanism on graph data. We evaluate our approach on two public logical reasoning machine reading comprehension(MRC) datasets: ReClor and LogiQA. Experiments on representative benchmark datasets demonstrate the effectiveness of our approach.

Information extraction from weakly structured radiological reports with natural language queries

ObjectivesProvide physicians and researchers an efficient way to extract information from weakly structured radiology reports with natural language processing (NLP) machine learning models.MethodsWe evaluate seven different German bidirectional encoder representations from transformers (BERT) models on a dataset of 857,783 unlabeled radiology reports and an annotated reading comprehension dataset in the format of SQuAD 2.0 based on 1223 additional reports.ResultsContinued pre-training of a BERT model on the radiology dataset and a medical online encyclopedia resulted in the most accurate model with an F1-score of 83.97% and an exact match score of 71.63% for answerable questions and 96.01% accuracy in detecting unanswerable questions. Fine-tuning a non-medical model without further pre-training led to the lowest-performing model. The final model proved stable against variation in the formulations of questions and in dealing with questions on topics excluded from the training set.ConclusionsGeneral domain BERT models further pre-trained on radiological data achieve high accuracy in answering questions on radiology reports. We propose to integrate our approach into the workflow of medical practitioners and researchers to extract information from radiology reports.Clinical relevance statementBy reducing the need for manual searches of radiology reports, radiologists’ resources are freed up, which indirectly benefits patients.Key Points• BERT models pre-trained on general domain datasets and radiology reports achieve high accuracy (83.97% F1-score) on question-answering for radiology reports.• The best performing model achieves an F1-score of 83.97% for answerable questions and 96.01% accuracy for questions without an answer.• Additional radiology-specific pretraining of all investigated BERT models improves their performance.Graphical

A deep-learning-based grading system (ASAG) for reading comprehension assessment by using aphorisms as open-answer-questions

Today reading comprehension is considered an essential skill in modern life, therefore, higher education students require more specific skills to understand, interpret and evaluate texts effectively. Short answer questions (SAQs) are one of the relevant and proper tools for assessing reading comprehension skills. Unlike multiple-choice questions, SAQs allow for the assessment of cognitive abilities such as attention, language, perception, and problem solving. However, the task of SAQs scoring is time-consuming and susceptible to ambiguity. Automatic Short Answer Grading (ASAG) is a new paradigm that could help solve these problems. This experimental analysis aims to implement ASAG using several approaches to sentence embedding based on deep learning with a multilayer perceptron regression layer on the top, trained with a reading comprehension dataset based on aphorisms. For experimental testing, the available dataset is composed of answers given by 199 undergraduate students in Spanish. BERT and Skip-Thought models are tested with different hyperparameters to find the best performance in terms of Pearson correlation coefficient and RMSE against human experts grades. The result of the current study showed that BERT model performed better than other approaches.

GAAMA 2.0: An Integrated System That Answers Boolean and Extractive Questions

Recent machine reading comprehension datasets include extractive and boolean questions but current approaches do not offer integrated support for answering both question types. We present a front-end demo to a multilingual machine reading comprehension system that handles boolean and extractive questions. It provides a yes/no answer and highlights the supporting evidence for boolean questions. It provides an answer for extractive questions and highlights the answer in the passage. Our system, GAAMA 2.0, achieved first place on the TyDi QA leaderboard at the time of submission. We contrast two different implementations of our approach: including multiple transformer models for easy deployment, and a shared transformer model utilizing adapters to reduce GPU memory footprint for a resource-constrained environment.

Leveraging Structured Information from a Passage to Generate Questions

Question Generation (QG) is the task of utilizing Artificial Intelligence (Al) technology to generate questions that can be answered by a span of text within a given passage. Existing research on QG in the educational field struggles with two challenges: the mainstream QG models based on seq-to-seq fail to utilize the structured information from the passage; the other is the lack of specialized educational QG datasets. To address the challenges, a specialized QG dataset, reading comprehension dataset from examinations for QG (named RACE4QG), is reconstructed by applying a new answer tagging approach and a data-filtering strategy to the RACE dataset. Further, an end-to-end QG model, which can exploit the intra- and inter-sentence information to generate better questions, is proposed. In our model, the encoder utilizes a Gated Recurrent Units (GRU) network, which takes the concatenation of word embedding, answer tagging, and Graph Attention neTworks(GAT) embedding as input. The hidden states of the GRU are operated with a gated self-attention to obtain the final passage-answer representation, which will be fed to the decoder. Results show that our model outperforms baselines on automatic metrics and human evaluation. Consequently, the model improves the baseline by 0.44, 1.32, and 1.34 on BLEU-4, ROUGE-L, and METEOR metrics, respectively, indicating the effectivity and reliability of our model. Its gap with human expectations also reflects the research potential.

A Feedback-Enhanced Two-Stage Framework for judicial machine reading comprehension

Machine Reading Comprehension (MRC) is the task of teaching machines to understand and answer questions based on a given text. In the judicial domain, MRC has garnered attention as a novel strategy for addressing a variety of legal issues. Judicial MRC generally requires interpretability, meaning that the model should not only answer the question correctly but also provide supporting evidence sentences. A straightforward approach is to handle the two tasks separately, which ignores the relation between the two tasks and leads to the loss of annotated information. To make better use of the limited judicial MRC annotation data, we simulate the strategies used by humans in solving MRC problems and propose the Feedback-Enhanced Two-Stage Framework for Machine Reading Comprehension (FETSF-MRC). This framework consists of two cascaded modules: (a) a scanning module that identifies evidence sentences and (b) a detailed reading module that focuses on the evidence to identify candidate answers and provides feedback to the scanning module. Experiments on two Chinese judicial reading comprehension datasets (CJRC, CAIL2020) and an open-domain English dataset HotpotQA show that FETSF-MRC achieves superior performance compared to several baselines. FETSF-MRC outperforms the best baseline by 1.12% and 1.49% in joint F1 score on the CJRC and CAIL2020 datasets, respectively, and achieves 73.74% joint F1 score on the HotpotQA.

An Automatic Question Generator for Chinese Comprehension

Question generation (QG) is a natural language processing (NLP) problem that aims to generate natural questions from a given sentence or paragraph. QG has many applications, especially in education. For example, QG can complement teachers’ efforts in creating assessment materials by automatically generating many related questions. QG can also be used to generate frequently asked question (FAQ) sets for business. Question answering (QA) can benefit from QG, where the training dataset of QA can be enriched using QG to improve the learning and performance of QA algorithms. However, most of the existing works and tools in QG are designed for English text. This paper presents the design of a web-based question generator for Chinese comprehension. The generator provides a user-friendly web interface for users to generate a set of wh-questions (i.e., what, who, when, where, why, and how) based on a Chinese text conditioned on a corresponding set of answer phrases. The web interface allows users to easily refine the answer phrases that are automatically generated by the web generator. The underlying question generation is based on the transformer approach, which was trained on a dataset combined from three publicly available Chinese reading comprehension datasets, namely, DRUD, CMRC2017, and CMRC2018. Linguistic features such as parts of speech (POS) and named-entity recognition (NER) are extracted from the text, which together with the original text and the answer phrases, are then fed into a machine learning algorithm based on a pre-trained mT5 model. The generated questions with answers are displayed in a user-friendly format, supplemented with the source sentences in the text used for generating each question. We expect the design of this web tool to provide insight into how Chinese question generation can be made easily accessible to users with low computer literacy.

PQuAD: A Persian question answering dataset

We present the Persian Question Answering Dataset (PQuAD), a crowdsourced reading comprehension dataset on Persian Wikipedia articles. It includes 80,000 questions along with their answers, with 25% of the questions being adversarially unanswerable. We examine various properties of the dataset to show the diversity and the level of its difficulty as a MRC benchmark. By releasing this dataset, we aim to ease research on Persian reading comprehension and the development of Persian question answering systems. Our experiments on different state-of-the-art pre-trained contextualized language models show 74.8% Exact Match (EM) and 87.6% F1-score that can be used as the baseline results for further research on Persian QA.

DIRECT: Toward Dialogue-Based Reading Comprehension Tutoring

A major challenge in education is to provide students with a personalized learning experience. This study aims to address this by developing a dialogue-based intelligent tutoring system (ITS) that imitates human expert tutors. The ITS asks questions, assesses student answers, provides hints, and even chats to encourage student engagement. We constructed the Dialogue-based Reading Comprehension Tutoring (DIRECT) dataset to simulate real-world pedagogical scenarios with the assessment labels and key sentences to support tutoring. The DIRECT dataset is based on RACE, which is a large-scale English reading comprehension dataset. In addition, we propose a neural pipeline approach to model the tutoring tasks and conduct a comprehensive analysis on the results, including a human evaluation. The results show that our model performs well in generating questions, assessing answers, and chatting, showing high potential although some challenges remain. The proposed model provides a good basis for further development of dialogue-based ITSs.

ViMRC - VLSP 2021: Using XLM-RoBERTa and Filter Output for Vietnamese Machine Reading Comprehension

Machine Reading Comprehension (MRC) has recently made significant progress. This paper is the result of our participation in building an MRC system specifically for Vietnamese on Vietnamese Machine Reading Comprehension at the 8th International Workshop on Vietnamese Language and Speech Processing (VLSP 2021). Based on SQuAD2.0, the organizing committee developed the Vietnamese Question Answering Dataset UIT-ViQuAD2.0, a reading comprehension dataset consisting of questions posed by crowd-workers on a set of Wikipedia Vietnamese articles. The UIT-ViQuAD2.0 dataset evolved from version 1.0 with the difference that version 2.0 contained answerable and unanswerable questions. The challenge of this problem is to distinguish between answerable and unanswerable questions. The answer to every question is a span of text, from the corresponding reading passage, or the question might be unanswerable. Our system employs simple yet highly effective methods. The system uses a pre-trained language model called XLM-RoBERTa (XLM-R), combined with filtering results from multiple output files to produce the final result. We created about 5-7 output files and select the answers with the most repetitions as the final prediction answer. After filtering, our system increased from 75.172% to 76.386% at the F1 measure and achieved 65,329% in the EM measure on the Private Test set.

AT-CRF: A Chinese Reading Comprehension Algorithm Based on Attention Mechanism and Conditional Random Fields

Machine reading comprehension (MRC) is an important research topic in the field of Natural Language Processing (NLP). However, traditional MRC models often face challenges of information loss, lack of capability to retain long-distance dependence, and inability to deal with unanswerable questions where answers are not available in given texts. In this paper, a Chinese reading comprehension algorithm, called the Attention and Conditional Random Filed (AT-CRF) Reader, is proposed to address the above challenges. Firstly, RoBERTa, a pre-trained language model, is introduced to obtain the embedding representations of input. Then, a depthwise separable convolution neural network and attention mechanisms are used to replace the recurrent neural network for encoding. Next, the attention flow and self-attention mechanisms are used to obtain the context–query internal relation. Finally, the conditional random field is used to handle unanswerable questions and predict the correct answer. Experiments were conducted on the two Chinese machine reading comprehension datasets, CMRC2018 and DuReader-checklist, and the results showed that, compared with the baseline model, the F1 scores achieved by our AT-CRF Reader model has improved by 2.65% and 2.68%, and the EM values increased by 4.45% and 3.88%.

Arabic machine reading comprehension on the Holy Qur’an using CL-AraBERT

In this work, we tackle the problem of machine reading comprehension (MRC) on the Holy Qur’an to address the lack of Arabic datasets and systems for this important task. We construct QRCD as the first Qur’anic Reading Comprehension Dataset, composed of 1,337 question-passage-answer triplets for 1,093 question-passage pairs, of which 14% are multi-answer questions. We then introduce CLassical-AraBERT (CL-AraBERT for short), a new AraBERT-based pre-trained model, which is further pre-trained on about 1.0B-word Classical Arabic (CA) dataset, to complement the Modern Standard Arabic (MSA) resources used in pre-training the initial model, and make it a better fit for the task. Finally, we leverage cross-lingual transfer learning from MSA to CA, and fine-tune CL-AraBERT as a reader using two MSA-based MRC datasets followed by our QRCD dataset to constitute the first (to the best of our knowledge) MRC system on the Holy Qur’an. To evaluate our system, we introduce Partial Average Precision (pAP) as an adapted version of the traditional rank-based Average Precision measure, which integrates partial matching in the evaluation over multi-answer and single-answer MSA questions. Adopting two experimental evaluation setups (hold-out and cross validation (CV)), we empirically show that the fine-tuned CL-AraBERT reader model significantly outperforms the baseline fine-tuned AraBERT reader model by 6.12 and 3.75 points in pAP scores, in the hold-out and CV setups, respectively. To promote further research on this task and other related tasks on Qur’an and Classical Arabic text, we make both the QRCD dataset and the pre-trained CL-AraBERT model publicly available.

ExpMRC: explainability evaluation for machine reading comprehension

Achieving human-level performance on some Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, it is necessary to provide both answer prediction and its explanation to further improve the MRC system's reliability, especially for real-life applications. In this paper, we propose a new benchmark called ExpMRC for evaluating the textual explainability of the MRC systems. ExpMRC contains four subsets, including SQuAD, CMRC 2018, RACE+, and C3, with additional annotations of the answer's evidence. The MRC systems are required to give not only the correct answer but also its explanation. We use state-of-the-art PLMs to build baseline systems and adopt various unsupervised approaches to extract both answer and evidence spans without human-annotated evidence spans. The experimental results show that these models are still far from human performance, suggesting that the ExpMRC is challenging. Resources (data and baselines) are available through https://github.com/ymcui/expmrc.

Multilingual multi-aspect explainability analyses on machine reading comprehension models

SummaryAchieving human-level performance on some of the machine reading comprehension (MRC) datasets is no longer challenging with the help of powerful pre-trained language models (PLMs). However, the internal mechanism of these artifacts remains unclear, placing an obstacle to further understand these models. This paper focuses on conducting a series of analytical experiments to examine the relations between the multi-head self-attention and the final MRC system performance, revealing the potential explainability in PLM-based MRC models. To ensure the robustness of the analyses, we perform our experiments in a multilingual way on top of various PLMs. We discover that passage-to-question and passage understanding attentions are the most important ones in the question answering process, showing strong correlations to the final performance than other parts. Through comprehensive visualizations and case studies, we also observe several general findings on the attention maps, which can be helpful to understand how these models solve the questions.

Constructing Arabic Reading Comprehension Datasets: Arabic WikiReading and KaifLematha

Constructing Arabic Reading Comprehension Datasets: Arabic WikiReading and KaifLematha