Answer Extraction Research Articles

Improving ranking-based question answering with weak supervision for low-resource Qur’anic texts

This work tackles the challenge of ranking-based machine reading comprehension (MRC), where a question answering (QA) system generates a ranked list of relevant answers for each question instead of simply extracting a single answer. We highlight the limitations of traditional learning methods in this setting, particularly under limited training data. To address these issues, we propose a novel ranking-inspired learning method that focuses on ranking multiple answer spans instead of single answer extraction. This method leverages lexical overlap as weak supervision to guide the ranking process. We evaluate our approach on the Qur’an Reading Comprehension Dataset (QRCD), a low-resource Arabic dataset over the Holy Qur’an. We employ transfer learning with external resources to fine-tune various transformer-based models, mitigating the low-resource challenge. Experimental results demonstrate that our proposed method significantly outperforms standard mechanisms across different models. Furthermore, we show its better alignment with the ranking-based MRC task and the effectiveness of external resources for this low-resource dataset. Our best performing model achieves a state-of-the-art partial Reciprocal Rank (pRR) score of 63.82%, surpassing the previous best-known score of 58.60%. To foster further research, we release code [GitHub repository:github.com/mohammed-elkomy/weakly-supervised-mrc], trained models, and detailed experiments to the community.

MuHeQA: Zero-shot question answering over multiple and heterogeneous knowledge bases

There are two main limitations in most of the existing Knowledge Graph Question Answering (KGQA) algorithms. First, the approaches depend heavily on the structure and cannot be easily adapted to other KGs. Second, the availability and amount of additional domain-specific data in structured or unstructured formats has also proven to be critical in many of these systems. Such dependencies limit the applicability of KGQA systems and make their adoption difficult. A novel algorithm is proposed, MuHeQA, that alleviates both limitations by retrieving the answer from textual content automatically generated from KGs instead of queries over them. This new approach (1) works on one or several KGs simultaneously, (2) does not require training data what makes it is domain-independent, (3) enables the combination of knowledge graphs with unstructured information sources to build the answer, and (4) reduces the dependency on the underlying schema since it does not navigate through structured content but only reads property values. MuHeQA extracts answers from textual summaries created by combining information related to the question from multiple knowledge bases, be them structured or not. Experiments over Wikidata and DBpedia show that our approach achieves comparable performance to other approaches in single-fact questions while being domain and KG independent. Results raise important questions for future work about how the textual content that can be created from knowledge graphs enables answer extraction.

Explanatory argument extraction of correct answers in resident medical exams

Developing technology to assist medical experts in their everyday decision-making is currently a hot topic in the field of Artificial Intelligence (AI). This is specially true within the framework of Evidence-Based Medicine (EBM), where the aim is to facilitate the extraction of relevant information using natural language as a tool for mediating in human–AI interaction. In this context, AI techniques can be beneficial in finding arguments for past decisions in evolution notes or patient journeys, especially when different doctors are involved in a patient’s care. In those documents the decision-making process towards treating the patient is reported. Thus, applying Natural Language Processing (NLP) techniques has the potential to assist doctors in extracting arguments for a more comprehensive understanding of the decisions made. This work focuses on the explanatory argument identification step by setting up the task in a Question Answering (QA) scenario in which clinicians ask questions to the AI model to assist them in identifying those arguments. In order to explore the capabilities of current AI-based language models, we present a new dataset which, unlike previous work: (i) includes not only explanatory arguments for the correct hypothesis, but also arguments to reason on the incorrectness of other hypotheses; (ii) the explanations are written originally in Spanish by doctors to reason over cases from the Spanish Residency Medical Exams. Furthermore, this new benchmark allows us to set up a novel extractive task by identifying the explanation written by medical doctors that supports the correct answer within an argumentative text. An additional benefit of our approach lies in its ability to evaluate the extractive performance of language models using automatic metrics, which in the Antidote CasiMedicos dataset corresponds to a 74.47 F1 score. Comprehensive experimentation shows that our novel dataset and approach is an effective technique to help practitioners in identifying relevant evidence-based explanations for medical questions.

Data or mathematics? Solutions to semantic problems in artificial intelligence

Data support is already driving the development of artificial intelligence. But it cannot solve the semantic problem of artificial intelligence. This requires improving the semantic understanding ability of artificial intelligence. Therefore, a question answering system based on semantic problem processing is proposed in this study. The question answering system utilizes an improved unsupervised method to extract keywords. This technology integrates the semantic feature information of text into traditional word graph model algorithms. On this basis, semantic similarity information is used to calculate and allocate the initial values and edge weights of each node in the PageRank model. And corresponding restart probability matrices and transition probability matrices are constructed for iterative calculation and keyword extraction. Simultaneously, an improved semantic dependency tree was utilized for answer extraction. The improved keyword extraction method shows a decreasing trend in P and R values. The improved answer extraction method has a maximum P-value of 0.876 in the training set and 0.852 in the test set. In a question answering system based on keyword and answer extraction, the improved method has lower loss function values and running time. The improved method has a larger area under ROC. The results of the validation analysis confirm that the improved method in this experiment has high accuracy and robustness when dealing with semantic problems.

Evaluating the performance of multilingual models in answer extraction and question generation

Multiple-choice test generation is one of the most complex NLP problems, especially in languages other than English, where there is a lack of prior research. After a review of the literature, it has been verified that some methods like the usage of rule-based systems or primitive neural networks have led to the application of a recent architecture, the Transformer architecture, in the tasks of Answer Extraction (AE) and Question Generation (QG). Thereby, this study is centred in searching and developing better models for the AE and QG tasks in Spanish, using an answer-aware methodology. For this purpose, three multilingual models (mT5-base, mT0-base and BLOOMZ-560 M) have been fine-tuned using three different datasets: a translation to Spanish of the SQuAD dataset; SQAC, which is a dataset in Spanish; and their union (SQuAD + SQAC), which shows slightly better results. Regarding the models, the performance of mT5-base has been compared with that found in two newer models, mT0-base and BLOOMZ-560 M. These models were fine-tuned for multiple tasks in literature, including AE and QG, but, in general, the best results are obtained from the mT5 models trained in our study with the SQuAD + SQAC dataset. Nonetheless, some other good results are obtained from mT5 models trained only with the SQAC dataset. For their evaluation, the widely used BLEU1-4, METEOR and ROUGE-L metrics have been obtained, where mT5 outperforms some similar research works. Besides, CIDEr, SARI, GLEU, WER and the cosine similarity metrics have been calculated to present a benchmark within the AE and QG problems for future work.

Document-Level Relation Extraction Based on Machine Reading Comprehension and Hybrid Pointer-sequence Labeling

Document-level relational extraction requires reading, memorization, and reasoning to discover relevant factual information in multiple sentences. It is difficult for the current hierarchical network and graph network methods to fully capture the structural information behind the document and make natural reasoning from the context. Different from the previous methods, this article reconstructs the relation extraction task into a machine reading comprehension task. Each pair of entities and relationships is characterized by a question template, and the extraction of entities and relationships is translated into identifying answers from the context. To enhance the context comprehension ability of the extraction model and achieve more precise extraction, we introduce large language models (LLMs) during question construction, enabling the generation of exemplary answers. Besides, to solve the multi-label and multi-entity problems in documents, we propose a new answer extraction model based on hybrid pointer-sequence labeling, which improves the reasoning ability of the model and realizes the extraction of zero or multiple answers in documents. Extensive experiments on three public datasets show that the proposed method is effective.

An Algorithm for Exploratory Analysis and Normalization of Big Data with Pandas

The digitization of business processes and the extraction of answers to user requests for big data are modern problems that are of great interest to scientists and researchers. The data generated so far, located in various corpora, is much more than can be analyzed. Therefore, they are collected, identified, cleaned and normalized to be used most adequately. Segmentation, assumptions and hypotheses contribute to the degree of satisfaction with the returned result. The research proposed a general method for collecting, cleaning and normalizing data from various sources, structurally modelling it into appropriate models, then testing hypotheses and analyzing the obtained results to conclude large academic data that will benefit the business in making management decisions. This is possible with the means of computational linguistics and with the help of Python data manipulation libraries.

Question answering over knowledge graphs using BERT based relation mapping

AbstractA knowledge graph (KG) is a structured form of knowledge describing real‐world entities, properties and relationships as a graph. Question answering over knowledge graphs (KGQA) allows people to ask questions in natural language and extract answers from KG accurately and more quickly. The main task of a KGQA is to convert a natural language query to the corresponding structured query form like SPARQL. However, generating the precise SPARQL query from a question is challenging and highly error‐prone. Here we propose a question‐answering framework that uses KG to answer simple questions without using SPARQL. Question classification, dependency parsing, entity linking, BERT‐based relation finding and answer extraction constitute the main modules of the approach. We have used the DBpedia as the KG and tested the end‐to‐end system with a subset of QALD‐4, LC‐QuAD and SimpleQuestions datasets. Results show considerable improvement compared to other approaches in terms of F1‐score.

Multimodal representative answer extraction in community question answering

To solve the information overload problem of multimodal answers in community question answering (CQA), this paper proposes a multimodal representative answer extraction method. First, the method of similarity calculation between multimodal answers is constructed, and multimodal clustering is used to cluster answers. Then, a binary multi-objective optimization model with three objective functions including multimodal answer coverage, multimodal answer redundancy, and multimodal answer consistency is constructed to extract a representative subset of answers. The improved Beluga whale optimization algorithm (MTRL-BWO), based on tent mapping, reinforcement learning, and multiple swarm strategy, is designed to increase the diversity of the population while avoiding local optima to improve the search capability and solution accuracy of the algorithm. Experimental results show the feasibility and superior performance of the proposed method.

The DistilBERT Model: A Promising Approach to Improve Machine Reading Comprehension Models

Machine Reading Comprehension (MRC) is a challenging task in the field of Natural Language Processing (NLP), where a machine is required to read a given text passage and answer a set of questions based on it. This paper provides an overview of recent advances in MRC and highlights some of the key challenges and future directions of this research area. It also evaluates the performance of several baseline models on the dataset, evaluates the challenges that the dataset poses for existing MRC models, and introduces the DistilBERT model to improve the accuracy of the answer extraction process. The supervised paradigm for training machine reading and comprehension models represents a practical path forward for creating comprehensive natural language understanding systems. To enhance the DistilBERT basic model's functionality, we have experimented with a variety of question heads that differ in the number of layers, activation function, and general structure. DistilBERT is a model for question-resolution tasks that is successful and delivers state-of-the-art performance while requiring less computational resources than large models like BERT, according to the presented technique. We could enhance the model's functionality and obtain a better understanding of how the model functions by investigating other question head architectures. These findings could serve as a foundation for future study on how to make question-and-answer systems and other tasks connected to the processing of natural languages.

So2al-wa-Gwab: A New Arabic Question-Answering Dataset Trained on Answer Extraction Models

Question answering (QA) is the task of responding to questions posed by users automatically. A question-answering system is divided into three main components: question analysis, information retrieval, and answer extraction. This paper has focused only on the answer extraction part. In the past couple of years, many QA systems have been developed and become mature and ready for use in different languages. Nevertheless, the advancement of Arabic QA systems still faces different obstacles and a lack of relevant resources and tools for researchers. This paper presents the So2al-wa-Gwab dataset since the publicly available datasets include various faults, such as the use of machine translation to build the data, a short context size, and a small number of question-answer pairings. Thus, this new dataset avoids the aforementioned drawbacks. Furthermore, in this paper, we have trained three deep learning models, namely, Bi-Directional flow network (BiDAF), QA Network (QANet), and BERT model, and tested them on seven different datasets, thus providing a comprehensive comparison between existing Arabic QA datasets. The obtained results emphasize that machine-translated datasets fall back when compared with human-annotated data. Also, the QA task becomes harder as the context, from which to extract the answer, becomes larger.

Building a deep learning-based QA system from a CQA dataset

A man-made machine-reading comprehension (MRC) dataset is necessary to train the answer extraction part of existing Question Answering (QA) systems. However, a high-quality and well-structured dataset with question-paragraph-answer pairs is not usually found in the real world. Furthermore, updating or building an MRC dataset is a challenging and costly affair. To address these shortcomings, we propose a QA system that uses a large-scale English Community Question Answering (CQA) dataset (i.e., Stack Exchange) composed of 3,081,834 question-answer pairs. The QA system adopts a classifier-retriever-summarizer structure design. The question classifier and the answer retriever part are based on a Bidirectional Encoder Representations from Transformers (BERT) Natural Language Processing (NLP) model by Google, and the summarizer part introduces a deep learning-based Text-to-Text Transfer Transformer (T5) model to summarize the long answers. We instantiated the proposed QA system with 140 topics from the CQA dataset (including topics such as biology, law, politics, etc.) and conducted human and automatic evaluations. Our system presented encouraging results, considering that it provides high-quality answers to the questions in the test set and satisfied the requirements to develop a QA system without MRC datasets. Our results show the potential of building automatic and high-performance QA systems without being limited by man-made datasets, a significant step forward in the research of open-domain or specific-domain QA systems.

MGRC: An End-to-End Multigranularity Reading Comprehension Model for Question Answering.

Deep neural network-based models have achieved great success in extractive question answering. Recently, many works have been proposed to model multistage matching for this task, which usually first retrieve relevant paragraphs or sentences and then extract an answer span from the retrieved results. However, such a pipeline-based approach suffers from the error propagation problem, especially for sentence-level retrieval that is usually difficult to achieve high accuracy due to the severe data imbalance problem. Furthermore, since the paragraph/sentence selector and the answer extractor are closely related, modeling them independently does not fully exploit the power of multistage matching. To solve these problems, we propose a novel end-to-end multigranularity reading comprehension model, which is a unified framework to explicitly model three matching granularities, including paragraph identification, sentence selection, and answer extraction. Our approach has two main advantages. First, the end-to-end approach alleviates the error propagation problem in both the training and inference phases. Second, the shared features in a unified model improve the learning of representations of different matching granularities. We conduct a comprehensive comparison on four large-scale datasets (SQuAD-open, NewsQA, SQuAD 2.0, and SQuAD Adversarial) and verify that the proposed approach outperforms both the vanilla BERT model and existing multistage matching approaches. We also conduct an ablation study and verify the effectiveness of the proposed components in our model structure.

A graph model with integrated pattern and query-based technique for extracting answer to questions in community question answering system

This paper presents an extension of the community question-answering (CQA) system we developed previously. This graph-based system method that builds ranked answers for related questions using nKullback–Leibler (KL) divergence. The process of extracting answers to questions in this work involves; question core, building question query, query-based answer extraction (QBAE), pattern-based answer extraction (PBAE), and combined answer extraction. The source data for this work were existing data from ResearchGate, a socio-academic networking website that provides researchers the platform to collaborate, ask questions, and offer answers to questions. The performance for answer extraction for 2786 questions shows that when 80% of patterns and keywords were considered, QBAE and PBAE extracted 2765 and 2766 correct answers, respectively, while the QBAE + PBAE method extracted 2782 correct answers. Also, when 90% of patterns and keywords were utilized, QBAE and PBAE extracted 2782 and 2784 correct answers, whereas the QBAE + PBAE method extracted 2786 correct answers. Our method was able to identify 229 questions without answers. Finally, the evaluation of our model reveals high-performance accuracy and precision.

Multi-passage extraction-based machine reading comprehension based on verification sorting

For traditional single-passage machine reading comprehension, the text data of a single passage does not well reflect the complexity of practical application scenarios. Many researchers have shifted their research goals to study multi-passage machine reading comprehension. To solve the problem of multi-passage machine reading comprehension, this paper proposes a unified multi-module end-to-end reading comprehension model for passage retrieval, answer extraction, and multi-answer verification ranking using a pre-trained model. In this paper, the passage retrieval module selects the passage fragments with the highest probability of survival, and the answer extraction component extracts the possible candidate answers in each passage. The multi-answer verification ranking component uses an attention mechanism to fuse multiple candidate answer feature representations, obtains the score of each candidate answer and selects the candidate answer with the highest score as the final answer. Finally, through experimental validation, the proposed model achieves scores of 49.59 and 46.28 on the evaluation metrics BLUE-4 and ROUGH-L on the DuReader dataset, and scores of 44.78 and 46.45 on the metrics BLUE-1 and ROUGH-L on the MS-MARCO dataset, respectively.

ViMRC - VLSP 2021: Improving Retrospective Reader for Vietnamese Machine Reading Comprehension

In recent years, there are multiple systems (eg. search engines and dialogue systems) that require machines to be able to read and understand human text to serve several tasks in application. Machine Reading Comprehension (MRC) has posed a challenge to the Natural Language Processing (NLP) community in teaching machines to understand the meaning of human text in order to answer questions provided. Specifically in this challenge, the dataset contains questions that can be unanswerable, otherwise the answers can be extracted from the given passages. To deal with this challenge, our works mainly based on a recent approach, known as Retrospective Reader, to confronting unanswerable questions. Additionally, we focuses on enhancing the ability of answer extraction by applying properly attention mechanism and improving the representation ability through semantic information. Besides, we also present an ensemble way to acquire significant improvement in results provided by single models. Our method achieves 1$^{st}$ place on Vietnamese MRC shared task at the $8^{th}$ International Workshop on Vietnamese Language and Speech Processing (VLSP) with F1-score of \textbf{0.77241} and exact match (EM) of \textbf{0.66137} on the private test phase. For research purpose, our source code is available at \url{https://github.com/NamCyan/MRC\_VLSP2021}

Analyzing Linguistic Features for Answer Re-Ranking of Why-Questions

Why-type non-factoid questions are ambiguous and involve variations in their answers. A challenge in returning one appropriate answer to user requires the process of appropriate answer extraction, re-ranking and validation. There are cases where the need is to understand the meaning and context of a document rather than finding exact words involved in question. The paper addresses this problem by exploring lexico-syntactic, semantic and contextual query-dependent features, some of which are based on deep learning frameworks to depict the probability of answer candidate being relevant for the question. The features are weighted by the score returned by ensemble ExtraTreesClassifier according to features importance. An answer re-ranker model is implemented that finds the highest ranked answer comprising largest value of feature similarity between question and answer candidate and thus achieving 0.64 Mean Reciprocal Rank (MRR). Further, answer is validated by matching the answer type of answer candidate and returns the highest ranked answer candidate with matched answer type to a user.

Automated question generation and question answering from Turkish texts

While exam-style questions are a fundamental educational tool serving a variety of purposes, manual construction of questions is a complex process that requires training, experience and resources. Automatic question generation (QG) techniques can be utilized to satisfy the need for a continuous supply of new questions by streamlining their generation. However, compared to automatic question answering (QA), QG is a more challenging task. In this work, we fine-tune a multilingual T5 (mT5) transformer in a multitask setting for QA, QG and answer extraction tasks using Turkish QA datasets. To the best of our knowledge, this is the first academic work that performs automated text-to-text question generation from Turkish texts. Experimental evaluations show that the proposed multitask setting achieves state-of-the-art Turkish question answering and question generation performance on TQuADv1, TQuADv2 datasets and XQuAD Turkish split. The source code and the pretrained models are available at https://github.com/obss/turkish question-generation.

A Question Answering Framework Based on Hybridization of Deep Learning and Semantic Web Techniques

The question answering (QA) system has been existing for several years. QA systems are divided into different processes such as question processing, document processing, paragraph extraction, answer extraction, question analysis, phrase mapping, disambiguation, query construction, querying the Knowledge Base (KB), and result ordering on user response respectively. Based on these processes, many models have been developed using approaches ranging from linguistic, statistical, and pattern matching. Popular models are Feedback, Refinement and Extended VocabularY Aggregation (FREyA), PowerAqua, SemSek, Semantic Interpretation of User Queries for QA on Interlinked Data (SINA), DEep Answers for maNy Naturally Asked questions (DEANNA), gAnswer, SemGraph, OKBQA (Open Knowledge Base and Question Answering) and Semantic Question Answering (SQA), for performance evaluation, these mostly focus on higher precision, recall, and/or F-measure. However, most of these models are constrained in the following operations: the combination of knowledge bases from different sources, formalism for knowledge representation to support interoperability, optimization of query construction and generation, ranking of responses, and support for set operations (union, sorting, comparison, and aggregation on user query) during query generation. This research proposes a hybrid of recurrent neural network and semantic-web-based question answering as the (RNNSQA) framework that combines heterogeneous knowledge sources and improves on state-of-the-art query generation mechanisms to allow for integration of comprehensive question-type operations, and set-based operations listed above. First, there would be a combination of techniques that is Natural Language Processing (NLP) and Recurrent Neural Network (RNN) techniques in classifying questions into types. Secondly, the semantic web technique is then employed in generating (SPARQL Protocol and RDF (Resource Description Framework) Query Language) SPARQL-based candidate queries. The result of this study is an enhanced QA framework with improved query translation and construction capability.

ARmed question answering system

SummaryQuestion answering, retrieving an exact answer to a question posed in natural language, is an issue which has widely been studied in the open domain over the last decades. This, however, remains a real challenge in the medical domain as most existing systems only support a limited amount of question and answer types. The problem with proposed methods for Arabic language in the medical domain is that there is often a conflict between the extracted answer and user's requirements. This conflict is related to ambiguity. Nevertheless, the method we propose has successfully tackled this problem. Thus, in this article, we introduce ARmed, a system for automatically answering medical questions for Arabic language. ARmed consists of corpora study, pre‐processing, question analysis, documents/passages retrieval, and answer extraction. Compared with the previous studies, ARmed has the potential to handle a large number of questions and answer types. The experimental results show that ARmed achieves interesting results.