Knowledge Base Question Answering Research Articles

Multi-Modal Validation and Domain Interaction Learning for Knowledge-Based Visual Question Answering

Multi-Modal Validation and Domain Interaction Learning for Knowledge-Based Visual Question Answering

Prompting Large Language Models with Knowledge-Injection for Knowledge-Based Visual Question Answering

Prompting Large Language Models with Knowledge-Injection for Knowledge-Based Visual Question Answering

GS-CBR-KBQA: Graph-structured case-based reasoning for knowledge base question answering

Knowledge Base Question Answering (KBQA) task is an important research direction in natural language processing. Due to the flexibility and ambiguity of natural language, users’ questions often have more complex query types and richer semantic information. To address this issue, this paper proposes the GS-CBR-KBQA model, a Case-Based Reasoning model tailored for KBQA to improve the semantic parsing accuracy and interpretability of natural language questions. The model integrates Knowledge-oriented Programming Language (KoPL) reasoning graphs with query information, employing a Graph Auto-Encoder and the RoBERTa pretrained language model for a highly effective case retrieval. This integration leads to a more robust knowledge retrieval and application approach, particularly innovative in capturing the relationships within KoPL graphs. The model addresses explicitly complex questions such as multi-hop reasoning and questions involving intricate entity relationships. Finally, our extensive experiments show that the model performs excellently in accuracy and F1 metrics on benchmark datasets such as WebQSP and ComplexWebQuestions, particularly in complex question-answering. The code of our model is available at https://anonymous.4open.science/r/GS-CBR-KBQA.

Caption matters: a new perspective for knowledge-based visual question answering

Caption matters: a new perspective for knowledge-based visual question answering

Candidate-Heuristic In-Context Learning: A new framework for enhancing medical visual question answering with LLMs

Medical Visual Question Answering (MedVQA) is designed to answer natural language questions related to medical images. Existing methods largely adopting the cross-modal pre-training and fine-tuning paradigm, face limitations in accuracy due to data scarcity and insufficient incorporation of extensive medical knowledge. Drawing inspiration from the Knowledge-Based Visual Question Answering (KB-VQA) domain, which leverages Large Language Models (LLMs) and external knowledge bases, we introduce the Candidate-Heuristic In-Context Learning (CH-ICL) framework, a novel approach that leverages LLMs augmented with external knowledge to directly enhance existing MedVQA models. Specifically, we collect a pathology terminology dictionary from a public digital pathology library as an external knowledge base and use it to train a knowledge scope discriminator, which helps identify the knowledge scope required to answer a question. Then, we employ existing MedVQA models to provide reliable answer candidates along with their confidence scores. Finally, the knowledge scope and candidates, combined with retrieved in-context exemplars, are aggregated into prompts for heuristically guiding LLMs in answer generation. Experimental results on the PathVQA, VQA-RAD, and SLAKE public benchmarks show state-of-the-art performance, with improvements of 1.91%, 1.88%, and 2.17% respectively over the baseline. Code and dataset are available at https://github.com/ecoxial2007/CH-ICL.

A Question Answering Chatbot for Gastric Cancer Patients After Curative Gastrectomy: Development and Evaluation of User Experience and Performance.

Postoperative gastric cancer patients have many questions about managing their daily lives with various symptoms and discomfort. This study aimed to develop a knowledge-based question answering (QA) chatbot for their self-management and to evaluate the user experience and performance of the chatbot. To support the chatbot's natural language processing, we analyzed QA texts from an online self-help group, clinical guidelines, and refined frequently asked questions related to gastric cancer. We developed a named entity classification with seven superconcepts, 4544 subconcepts, and 1415 synonyms. We also developed a knowledge base by linking the users' classified question intents with the experts' answers and knowledge resources, including 677 question intents and scripts with standard QA pairs and similar question phrases. A chatbot called "GastricFAQ" was built, reflecting the question topics of the named entity classification and QA pairs of the knowledge base. User experience evaluation (N = 56) revealed the highest mean score for usefulness (4.41/5.00), with all other items rated 4.00 or higher, except desirability (3.85/5.00). The chatbot's accuracy, precision, recall, and F score ratings were 85.2%, 87.6%, 96.8%, and 92.0%, respectively, with immediate answers. GastricFAQ could be provided as one option to obtain immediate information with relatively high accuracy for postoperative gastric cancer patients.

Improving Topic Tracing with a Textual Reader for Conversational Knowledge Based Question Answering

Improving Topic Tracing with a Textual Reader for Conversational Knowledge Based Question Answering

FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering

Knowledge base question answering (KBQA) is a critical yet challenging task due to the vast number of entities within knowledge bases and the diversity of natural language questions posed by users. Unfortunately, the performance of most KBQA models tends to decline significantly in real-world scenarios where high-quality annotated data is insufficient. To mitigate the burden associated with manual annotation, we introduce FlexKBQA by utilizing Large Language Models (LLMs) as program translators for addressing the challenges inherent in the few-shot KBQA task. Specifically, FlexKBQA leverages automated algorithms to sample diverse programs, such as SPARQL queries, from the knowledge base, which are subsequently converted into natural language questions via LLMs. This synthetic dataset facilitates training a specialized lightweight model for the KB. Additionally, to reduce the barriers of distribution shift between synthetic data and real user questions, FlexKBQA introduces an executionguided self-training method to iterative leverage unlabeled user questions. Furthermore, we explore harnessing the inherent reasoning capability of LLMs to enhance the entire framework. Consequently, FlexKBQA delivers substantial flexibility, encompassing data annotation, deployment, and being domain agnostic. Through extensive experiments on GrailQA, WebQSP, and KQA Pro, we observe that under the few-shot even the more challenging zero-shot scenarios, FlexKBQA achieves impressive results with a few annotations, surpassing all previous baselines and even approaching the performance of supervised models, achieving a remarkable 93% performance relative to the fully-supervised models. We posit that FlexKBQA represents a significant advancement towards exploring better integration of large and lightweight models. Code is available at https://github.com/leezythu/FlexKBQA.

Code-Style In-Context Learning for Knowledge-Based Question Answering

Current methods for Knowledge-Based Question Answering (KBQA) usually rely on complex training techniques and model frameworks, leading to many limitations in practical applications. Recently, the emergence of In-Context Learning (ICL) capabilities in Large Language Models (LLMs) provides a simple and training-free semantic parsing paradigm for KBQA: Given a small number of questions and their labeled logical forms as demo examples, LLMs can understand the task intent and generate the logic form for a new question. However, current powerful LLMs have little exposure to logic forms during pre-training, resulting in a high format error rate. To solve this problem, we propose a code-style in-context learning method for KBQA, which converts the generation process of unfamiliar logical form into the more familiar code generation process for LLMs. Experimental results on three mainstream datasets show that our method dramatically mitigated the formatting error problem in generating logic forms while realizing a new SOTA on WebQSP, GrailQA, and GraphQ under the few-shot setting. The code and supplementary files are released at https://github.com/Arthurizijar/KB-Coder.

Prompting large language model with context and pre-answer for knowledge-based VQA

Existing studies apply Large Language Model (LLM) to knowledge-based Visual Question Answering (VQA) with encouraging results. Due to the insufficient input information, the previous methods still have shortcomings in constructing the prompt for LLM, and cannot fully activate the capacity of LLM. In addition, previous works adopt GPT-3 for inference, which has expensive costs. In this paper, we propose PCPA: a framework that Prompts LLM with Context and Pre-Answer for VQA. Specifically, we adopt a vanilla VQA model to generate in-context examples and candidate answers, and add a pre-answer selection layer to generate pre-answers. We integrate in-context examples and pre-answers into the prompt to inspire the LLM. In addition, we choose LLaMA instead of GPT-3, which is an open and free model. We build a small dataset to fine-tune the LLM. Compared to existing baselines, the PCPA improves accuracy by more than 2.1 and 1.5 on OK-VQA and A-OKVQA, respectively.

Coordinating explicit and implicit knowledge for knowledge-based VQA

Pre-trained models often generate plausible looking statements that are factually incorrect because of the inaccurate implicit knowledge contained in the model’s parameters. Related methods retrieve explicit knowledge from the external knowledge source to help improve the prediction performance and reliability. However, these methods often use weak training signals for the retriever, and require the model to make each prediction based on the retrieved knowledge, even when the retrieved knowledge is not reliable or the model can produce better prediction only using its implicit knowledge. Therefore, it is necessary to enable the pre-trained model to actively select more beneficial knowledge for producing better prediction. This work proposes a novel method to help the model to Coordinate Explicit and Implicit Knowledge (CEIK) for the knowledge-based visual question answering (VQA) task, which is an important direction of pre-trained models. Furthermore, a better training signal is proposed for the retriever according to whether the retrieved knowledge can correct the prediction. Experimental results demonstrate the effectiveness of our method.

Enhancing SPARQL Query Generation for Knowledge Base Question Answering Systems by Learning to Correct Triplets

Generating SPARQL queries from natural language questions is challenging in Knowledge Base Question Answering (KBQA) systems. The current state-of-the-art models heavily rely on fine-tuning pretrained models such as T5. However, these methods still encounter critical issues such as triple-flip errors (e.g., (subject, relation, object) is predicted as (object, relation, subject)). To address this limitation, we introduce TSET (Triplet Structure Enhanced T5), a model with a novel pretraining stage positioned between the initial T5 pretraining and the fine-tuning for the Text-to-SPARQL task. In this intermediary stage, we introduce a new objective called Triplet Structure Correction (TSC) to train the model on a SPARQL corpus derived from Wikidata. This objective aims to deepen the model’s understanding of the order of triplets. After this specialized pretraining, the model undergoes fine-tuning for SPARQL query generation, augmenting its query-generation capabilities. We also propose a method named “semantic transformation” to fortify the model’s grasp of SPARQL syntax and semantics without compromising the pre-trained weights of T5. Experimental results demonstrate that our proposed TSET outperforms existing methods on three well-established KBQA datasets: LC-QuAD 2.0, QALD-9 plus, and QALD-10, establishing a new state-of-the-art performance (95.0% F1 and 93.1% QM on LC-QuAD 2.0, 75.85% F1 and 61.76% QM on QALD-9 plus, 51.37% F1 and 40.05% QM on QALD-10).

DSAMR: Dual-Stream Attention Multi-hop Reasoning for knowledge-based visual question answering

Knowledge-based visual question answering aims to associate external knowledge facts for answering questions about images. Most existing methods emphasize high-order associations between knowledge facts and questions, and fail to consider the negative effects of unnecessary knowledge facts in multi-hop reasoning. In this paper, we propose a Dual-Stream Attention Multi-hop Reasoning (DSAMR) architecture that constructs two different attention streams to mitigate unnecessary knowledge facts. This dual-stream mechanism enables the model to reduce the attention weights on unnecessary knowledge while gathering essential knowledge by learning the implicit correlations between knowledge facts and questions. In addition, we designed a hypergraph knowledge extraction module in the architecture to extract optimal knowledge facts by evaluating the relevance of each knowledge fact to the question. The experimental results demonstrate the effectiveness of our method not only on the knowledge-based visual question answering dataset KVQA, but also on the multi-hop question answering dataset PathQuestion.

A Novel Joint Training Model for Knowledge Base Question Answering

A Novel Joint Training Model for Knowledge Base Question Answering

Knowledge enhancement and scene understanding for knowledge-based visual question answering

Knowledge enhancement and scene understanding for knowledge-based visual question answering

Knowledge-based Visual Question Answering about Named Entities

This thesis is positioned at the intersection of several research fields, Natural Language Processing, Information Retrieval (IR) and Computer Vision, which have unified around representation learning and pre-training methods. We have defined and studied a new multimodal task: Knowledge-based Visual Question Answering about Named Entities (KVQAE). We were particularly interested in cross-modal interactions and different ways of representing named entities. We also focused on data used to train and, more importantly, evaluate Question Answering systems through different metrics. We annotated a dataset for this purpose, the first in KVQAE comprising various types of entities. We also defined an experimental framework for dealing with KVQAE in two stages through an unstructured knowledge base and identified IR as the main bottleneck of KVQAE, especially for questions about non-person entities. To improve the IR stage, we studied different multimodal fusion methods, which are pre-trained through an original task: the Multimodal Inverse Cloze Task. We found that these models leveraged a cross-modal interaction that we had not originally considered, and which may address the heterogeneity of visual representations of named entities. These results were strengthened by a study of the CLIP model, which allows this cross-modal interaction to be modeled directly. Awarded by : Université Paris-Saclay, Orsay, France on 8 November 2023. Supervised by : Olivier Ferret and Camille Guinaudeau. Available at : https://www.theses.fr/s247993.

Complex Knowledge Base Question Answering: A Survey

Knowledge base question answering (KBQA) aims to answer a question over a knowledge base (KB). Early studies mainly focused on answering simple questions over KBs and achieved great success. However, their performances on complex questions are still far from satisfactory. Therefore, in recent years, researchers propose a large number of novel methods, which looked into the challenges of answering complex questions. In this survey, we review recent advances in KBQA with the focus on solving complex questions, which usually contain multiple subjects, express compound relations, or involve numerical operations. In detail, we begin with introducing the complex KBQA task and relevant background. Then, we present two mainstream categories of methods for complex KBQA, namely semantic parsing-based (SP-based) methods and information retrieval-based (IR-based) methods. Specifically, we illustrate their procedures with flow designs and discuss their difference and similarity. Next, we summarize the challenges that these two categories of methods encounter when answering complex questions, and explicate advanced solutions as well as techniques used in existing work. After that, we discuss the potential impact of pre-trained language models (PLMs) on complex KBQA. To help readers catch up with SOTA methods, we also provide a comprehensive evaluation and resource about complex KBQA task. Finally, we conclude and discuss several promising directions related to complex KBQA for future research.

SPARQL-QA-v2 system for Knowledge Base Question Answering

Accessing the large volumes of information available in public knowledge bases might be complicated for those users unfamiliar with formal languages, such as the SPARQL query language and the ontology definition languages. This issue can be overcome by providing systems able to answer questions posed in natural language on a knowledge base, a task that is called Knowledge Base Question Answering (KBQA) in the literature. More in detail, many KBQA systems aim at translating automatically questions into the corresponding SPARQL queries to be executed over the knowledge base to get the answers. Effective state-of-the-art KBQA systems are based on neural-machine translation but easily fail to recognize words that are Out Of the Vocabulary (OOV) of the training set. This is a serious issue while querying large ontologies where the list of entities is huge and easily evolves over time. In this paper, we present the SPARQL-QA-v2 system that combines in an innovative way Named Entity Linking, Named Entity Recognition, and Neural Machine Translation for addressing the problem of generating SPARQL queries from questions posed in natural language. We demonstrate empirically that SPARQL-QA-v2 is effective and resilient to OOV words and delivers state-of-the-art performance in well-known datasets for question answering over DBpedia and Wikidata knowledge bases.

Multigranularity Pruning Model for Subject Recognition Task under Knowledge Base Question Answering When General Models Fail

In general knowledge base question answering (KBQA) models, subject recognition (SR) is usually a precondition of finding an answer, and it is a common way to employ a general named entity recognition (NER) model such as BERT-CRF to recognize the subject. However, in previous researches, the difference between a NER task and a SR task is usually ignored, and a wrong entity recognized by the NER model will certainly lead to a wrong answer in the KBQA task, which is one bottleneck for KBQA performance. In this paper, a multigranularity pruning model (MGPM) is proposed to answer a question when general models fail to recognize a subject. In MGPM, the set of all possible subjects in the Knowledge Base (KB) is pruned by 4 multigranularity pruning submodels successively based on the constraint of relation (domain and tuple), string similarity, and semantic similarity. Experimental results show that our model is compatible with various KBQA models for both single-relation and complex questions answering. The integrated MGPM model (with the BERT-CRF model) achieves a SR accuracy of 94.4% on the SimpleQuestions dataset, 68.6% on the WebQuestionsSP dataset, and 63.7% on the WebQuestions dataset, which outperforms the original model by a margin of 3.6%, 8.6%, and 5.3%, respectively.

Cross-modality Multiple Relations Learning for Knowledge-based Visual Question Answering

Knowledge-based visual question answering not only needs to answer the questions based on images but also incorporates external knowledge to study reasoning in the joint space of vision and language. To bridge the gap between visual content and semantic cues, it is important to capture the question-related and semantics-rich vision-language connections. Most existing solutions model simple intra-modality relation or represent cross-modality relation using a single vector, which makes it difficult to effectively model complex connections between visual features and question features. Thus, we propose a cross-modality multiple relations learning model, aiming to better enrich cross-modality representations and construct advanced multi-modality knowledge triplets. First, we design a simple yet effective method to generate multiple relations that represent the rich cross-modality relations. The various cross-modality relations link the textual question to the related visual objects. These multi-modality triplets efficiently align the visual objects and corresponding textual answers. Second, to encourage multiple relations to better align with different semantic relations, we further formulate a novel global-local loss. The global loss enables the visual objects and corresponding textual answers close to each other through cross-modality relations in the vision-language space, and the local loss better preserves semantic diversity among multiple relations. Experimental results on the Outside Knowledge VQA and Knowledge-Routed Visual Question Reasoning datasets demonstrate that our model outperforms the state-of-the-art methods.