Answering System Research Articles

Experimental Design of Extractive Question-Answering Systems: Influence of Error Scores and Answer Length

Question-answering (QA) systems are becoming more and more important because they enable human-computer communication in a natural language. In recent years, significant progress has been made with transformer-based models that leverage deep learning in combination with large amounts of text data. However, a significant challenge with QA systems lies in their complexity rooted in the ambiguity and flexibility of a natural language. This makes even their evaluation a formidable task. For this reason, in this study, we focus on the evaluation of extractive question-answering (EQA) systems by conducting a large-scale analysis of distilBERT using benchmark data provided by the Stanford Question Answering Dataset (SQuAD). Specifically, the main objectives of this paper are fourfold. First, we study the influence of the answer length on the performance and we demonstrate that there is an inverse correlation between both. Second, we study differences in exact match (EM) measures because there are different definitions commonly used in the literature. As a result, we find that despite the fact that all of those measures are named ”exact match” these measures are actually different from each other. Third, we study the practical relevance of these different definitions because due to the ambivalent meaning of ”exact match” in the literature, it is often unclear if reported improvements are genuine or only due to a change in the exact match measure. Importantly, our results show that differences between differently defined EM measures are in the same order of magnitude as reported differences found in the literature. This raises concerns about the robustness of reported results. Fourth, we provide guidelines to improve the experimental design of general EQA studies, aiming to enhance performance evaluation and minimize the potential for spurious results.

Creating and validating the Fine-Grained Question Subjectivity Dataset (FQSD): A new benchmark for enhanced automatic subjective question answering systems.

In the domain of question subjectivity classification, there exists a need for detailed datasets that can foster advancements in Automatic Subjective Question Answering (ASQA) systems. Addressing the prevailing research gaps, this paper introduces the Fine-Grained Question Subjectivity Dataset (FQSD), which comprises 10,000 questions. The dataset distinguishes between subjective and objective questions and offers additional categorizations such as Subjective-types (Target, Attitude, Reason, Yes/No, None) and Comparison-form (Single, Comparative). Annotation reliability was confirmed via robust evaluation techniques, yielding a Fleiss's Kappa score of 0.76 and Pearson correlation values up to 0.80 among three annotators. We benchmarked FQSD against existing datasets such as (Yu, Zha, and Chua 2012), SubjQA (Bjerva 2020), and ConvEx-DS (Hernandez-Bocanegra 2021). Our dataset excelled in scale, linguistic diversity, and syntactic complexity, establishing a new standard for future research. We employed visual methodologies to provide a nuanced understanding of the dataset and its classes. Utilizing transformer-based models like BERT, XLNET, and RoBERTa for validation, RoBERTa achieved an outstanding F1-score of 97%, confirming the dataset's efficacy for the advanced subjectivity classification task. Furthermore, we utilized Local Interpretable Model-agnostic Explanations (LIME) to elucidate model decision-making, ensuring transparent and reliable model predictions in subjectivity classification tasks.

Elasto-Plastic Constitutive Behaviour Prediction and Uncertainty Quantification of Damaged Composite Structure under High-Strain Rate Loading

Abstract The current research assesses the consequences of various damages (crack and delamination) and high-strain loading conditions on the fibre-reinforced composite structure's elasto-plastic stress-strain (EPSS) characteristics. The constitutive responses are obtained numerically (finite element discretization) using higher-order polynomials (to count the deformation) with the help of the MATLAB platform. The EPSS responses are evaluated via the modified Ludwik equation and Cowper Symonds model under high strain rate loadings. The model accuracy has been tested through the comparison between the present numerical and the published experimental data available in open domain. Furthermore, various numerical examples present the effect of damages (debond and/or crack), forces and stress-strain characteristics for a wide range of strain rates (ranging from 0.001 s−1 to 50 s−1). The stochastic constitutive model is obtained to show the modelling importance and the corresponding analysis parameters including the uncertainty quantification. Finally, a detailed understanding of the damages and geometries on the polymeric composite structure are enumerated via the delve model for the futuristic design.

Domain-specific Embeddings for Question-Answering Systems: FAQs for Health Coaching

FAQs are widely used to respond to users’ knowledge needs within knowledge domains. While LLM might be a promising way to address user questions, they are still prone to hallucinations i.e., inaccurate or wrong responses, which, can, inter alia, lead to massive problems, including, but not limited to, ethical issues. As a part of the healthcare coach chatbot for young Nigerian HIV clients, the need to meet their information needs through FAQs is one of the main coaching requirements. In this paper, we explore if domain knowledge in HIV FAQs can be represented as text embeddings to retrieve similar questions matching user queries, thus improving the understanding of the chatbot and the satisfaction of the users. Specifically, we describe our approach to developing an FAQ chatbot for the domain of HIV. We used a predefined FAQ question-answer knowledge base in English and Pidgin co-created by HIV clients and experts from Nigeria and Switzerland. The results of the post-engagement survey show that the chatbot mostly understood the user’s questions and could identify relevant matching questions and retrieve an appropriate response.

ChatFFA: An ophthalmic chat system for unified vision-language understanding and question answering for fundus fluorescein angiography

Existing automatic analysis of fundus fluorescein angiography (FFA) images faces limitations, including a predetermined set of possible image classifications and being confined to text-based question-answering (QA) approaches. This study aims to address these limitations by developing an end-to-end unified model that utilizes synthetic data to train a visual question-answering model for FFA images. To achieve this, we employed ChatGPT to generate 4,110,581 QA pairs for a large FFA dataset, which encompassed a total of 654,343 FFA images from 9,392 participants. We then fine-tuned the Bootstrapping Language-Image Pre-training (BLIP) framework to enable simultaneous handling of vision and language. The performance of the fine-tuned model (ChatFFA) was thoroughly evaluated through automated and manual assessments, as well as case studies based on an external validation set, demonstrating satisfactory results. In conclusion, our ChatFFA system paves the way for improved efficiency and feasibility in medical imaging analysis by leveraging generative large language models.

Question answering system with text mining and deep networks

Question answering system with text mining and deep networks

Scoring Multi-hop Question Decomposition Using Masked Language Models

Question answering (QA) is a sub-field of Natural Language Processing (NLP) that focuses on developing systems capable of answering natural language queries. Within this domain, multi-hop question answering represents an advanced QA task that requires gathering and reasoning over multiple pieces of information from diverse sources or passages. To handle the complexity of multi-hop questions, question decomposition has been proven to be a valuable approach. This technique involves breaking down complex questions into simpler sub-questions, reducing the complexity of the problem. However, it’s worth noting that existing question decomposition methods often rely on training data, which may not always be readily available for low-resource languages or specialized domains. To address this issue, we propose a novel approach that utilizes pre-trained masked language models to score decomposition candidates in a zero-shot manner. The method involves generating decomposition candidates, scoring them using a pseudo-log likelihood estimation, and ranking them based on their scores. To evaluate the efficacy of the decomposition process, we conducted experiments on two datasets annotated on decomposition in two different languages, Arabic and English. Subsequently, we integrated our approach into a complete QA system and conducted a reading comprehension performance evaluation on the HotpotQA dataset. The obtained results emphasize that while the system exhibited a small drop in performance, it still maintained a significant advance compared to the baseline model. The proposed approach highlights the efficiency of the language model scoring technique in complex reasoning tasks such as multi-hop question decomposition.

An automated information extraction system from the knowledge graph based annual financial reports

This article presents a semantic web-based solution for extracting the relevant information automatically from the annual financial reports of the banks/financial institutions and presenting this information in a queryable form through a knowledge graph. The information in these reports is significantly desired by various stakeholders for making key investment decisions. However, this information is available in an unstructured format making it much more complex and challenging to understand and query manually or even through digital systems. Another challenge that makes the understanding of information more complex is the variation of terminologies among financial reports of different banks or financial institutions. The solution presented in this article signifies an ontological approach to solving the standardization problems of the terminologies in this domain. It further addresses the issue of semantic differences to extract relevant data sharing common semantics. Such semantics are then incorporated by implementing their representation as a Knowledge Graph to make the information understandable and queryable. Our results highlight the usage of Knowledge Graph in search engines, recommender systems and question-answering (Q-A) systems. This financial knowledge graph can also be used to serve the task of financial storytelling. The proposed solution is implemented and tested on the datasets of various banks and the results are presented through answers to competency questions evaluated on precision and recall measures.

BCSLinker: automatic method for constructing a knowledge graph of venous thromboembolism based on joint learning.

Venous thromboembolism (VTE) is characterized by high morbidity, mortality, and complex treatment. A VTE knowledge graph (VTEKG) can effectively integrate VTE-related medical knowledge and offer an intuitive description and analysis of the relations between medical entities. However, current methods for constructing knowledge graphs typically suffer from error propagation and redundant information. In this study, we propose a deep learning-based joint extraction model, Biaffine Common-Sequence Self-Attention Linker (BCSLinker), for Chinese electronic medical records to address the issues mentioned above, which often occur when constructing a VTEKG. First, the Biaffine Common-Sequence Self-Attention (BCsSa) module is employed to create global matrices and extract entities and relations simultaneously, mitigating error propagation. Second, the multi-label cross-entropy loss is utilized to diminish the impact of redundant information and enhance information extraction. We used the electronic medical record data of VTE patients from a tertiary hospital, achieving an F1 score of 86.9% on BCSLinker. It outperforms the other joint entity and relation extraction models discussed in this study. In addition, we developed a question-answering system based on the VTEKG as a structured data source. This study has constructed a more accurate and comprehensive VTEKG that can provide reference for diagnosing, evaluating, and treating VTE as well as supporting patient self-care, which is of considerable clinical value.

Interactive Question Answering Systems: Literature Review

Question-answering systems are recognized as popular and frequently effective means of information seeking on the web. In such systems, information seekers can receive a concise response to their queries by presenting their questions in natural language. Interactive question answering is a recently proposed and increasingly popular solution that resides at the intersection of question answering and dialogue systems . On the one hand, the user can ask questions in normal language and locate the actual response to her inquiry; on the other hand, the system can prolong the question-answering session into a dialogue if there are multiple probable replies, very few, or ambiguities in the initial request. By permitting the user to ask more questions, interactive question answering enables users to interact with the system and receive more precise results dynamically. This survey offers a detailed overview of the interactive question-answering methods that are prevalent in current literature. It begins by explaining the foundational principles of question-answering systems, hence defining new notations and taxonomies to combine all identified works inside a unified framework. The reviewed published work on interactive question-answering systems is then presented and examined in terms of its proposed methodology, evaluation approaches, and dataset/application domain. We also describe trends surrounding specific tasks and issues raised by the community, so shedding light on the future interests of scholars. Our work is further supported by a GitHub page synthesizing all the major topics covered in this literature study. https://sisinflab.github.io/interactive-question-answering-systems-survey/

Research on the Construction Method of Knowledge Base of Automatic Question Answering System for Labor Law

At present, with the rapid development of China's economy, labor relations in society are becoming more and more tense. Therefore, it is urgent and necessary to develop an automatic question and answer system for labor law to solve problems in legislation, law enforcement, justice, rights protection, etc., and the construction of labor law knowledge base is the core system of the automatic question and answer system of labor law. With the development and training of large AI models, the automatic question answering system is becoming more and more mature, and this paper will use FastGPT to build a local knowledge base and build a knowledge base of the automatic question answering system for labor law.

Unknown-class recognition adversarial network for open set domain adaptation fault diagnosis of rotating machinery

Unknown-class recognition adversarial network for open set domain adaptation fault diagnosis of rotating machinery

A Graph-Based Keyword Extraction Method for Academic Literature Knowledge Graph Construction

In this paper, we construct an academic literature knowledge graph based on the relationship between documents to facilitate the storage and research of academic literature data. Keywords are an important type of node in the knowledge graph. To solve the problem that there are no keywords in some documents for several reasons in the process of knowledge graph construction, an improved keyword extraction algorithm called TP-CoGlo-TextRank is proposed by using word frequency, position, word co-occurrence frequency, and a word embedding model. By combining the word frequency and position in the document, the importance of words is distinguished. By introducing the GloVe word-embedding model, which brings the external knowledge of documents into the TextRank algorithm, and combining the internal word co-occurrence frequency in the documents, the word-adjacency relationship is transferred non-uniformly. Finally, the words with the highest scores are combined into phrases if they are adjacent in the original text. The validity of the TP-CoGlo-TextRank algorithm is verified by experiments. On this basis, the Neo4j graph database is used to store and display the academic literature knowledge graph, to provide data support for research tasks such as text clustering, automatic summarization, and question-answering systems.

NEUROSYNTH: Enhancing Cognitive Computing using Deep Neural Networks

This study presents an innovative approach to enhance question-answering (QA) systems that utilize a RoBERTa-based architecture and complexity-enhanced input features. The work is divided into four primary parts: training methodology, feature engineering, building models, and data preprocessing. We propose a Python function that uses readability measures and natural language processing techniques to calculate the linguistic difficulty metrics for input sentences. TensorFlow Datasets (TFDS) are then used to load and preprocess the SquAD (Stanford Question Answering Dataset) dataset to enable effective training. Word embeddings from previously trained GloVe vectors are integrated with complexity metrics to prepare input features that add contextual information to the input representation. With the inclusion of the enriched features, a distinctive question-answering model based on the RoBERTa architecture is trained using the AdamW optimizer and CrossEntropyLoss. Iterative epochs are used in the training process to optimize the model's parameters and minimize the loss function. An independent validation dataset is used to evaluate the model's performance, proving the usefulness of the suggested method in improving the accuracy and robustness of the QA system. All in all, this work offers an organized strategy for improving the quality of systems by fusing cutting-edge neural architecture with input properties that are increased by complexity.

Large Language Model-Based Evaluation of Medical Question Answering Systems: Algorithm Development and Case Study.

Healthcare systems are increasingly resource constrained, leaving less time for important patient-provider interactions. Conversational agents (CAs) could be used to support the provision of information and to answer patients' questions. However, information must be accessible to a variety of patient populations, which requires understanding questions expressed at different language levels. This study describes the use of Large Language Models (LLMs) to evaluate predefined medical content in CAs across patient populations. These simulated populations are characterized by a range of health literacy. The evaluation framework includes both fully automated and semi-automated procedures to assess the performance of a CA. A case study in the domain of mammography shows that LLMs can simulate questions from different patient populations. However, the accuracy of the answers provided varies depending on the level of health literacy. Our scalable evaluation framework enables the simulation of patient populations with different health literacy levels and helps to evaluate domain specific CAs, thus promoting their integration into clinical practice. Future research aims to extend the framework to CAs without predefined content and to apply LLMs to adapt medical information to the specific (health) literacy level of the user.

Ecological aspects and relationships of the emblematic Vachellia spp. exposed to anthropic pressures and parasitism in natural hyper-arid ecosystems: ethnobotanical elements, morphology, and biological nitrogen fixation

Main conclusionEmblematic Vachellia spp. naturally exposed to hyper-arid conditions, intensive grazing, and parasitism maintain a high nitrogen content and functional mutualistic nitrogen-fixing symbioses.AlUla region in Saudi Arabia has a rich history regarding mankind, local wildlife, and fertility islands suitable for leguminous species, such as the emblematic Vachellia spp. desert trees. In this region, we investigated the characteristics of desert legumes in two nature reserves (Sharaan and Madakhil), at one archaeological site (Hegra), and in open public domains at Al. Ward and Jabal Abu Oud. Biological nitrogen fixation (BNF), isotopes, and N and C contents were investigated through multiple lenses, including parasitism, plant tissues, species identification, plant maturity, health status, and plant growth. The average BNF rates of 19 Vachellia gerrardii and 21 Vachellia tortilis trees were respectively 39 and 67%, with low signs of inner N content fluctuations (2.10–2.63% N) compared to other co-occurring plants. The BNF of 23 R. raetam was just as high, with an average of 65% and steady inner N contents of 2.25 ± 0.30%. Regarding parasitism, infected Vachellia trees were unfazed compared to uninfected trees, thereby challenging the commonly accepted detrimental role of parasites. Overall, these results suggest that Vachellia trees and R. raetam shrubs exploit BNF in hyper-arid environments to maintain a high N content when exposed to parasitism and grazing. These findings underline the pivotal role of plant-bacteria mutualistic symbioses in desert environments. All ecological traits and relationships mentioned are further arguments in favor of these legumes serving as keystone species for ecological restoration and agro-silvo-pastoralism in the AlUla region.Graphical abstract

CLOSED-DOMAIN QUESTION-ANSWERING SYSTEM FOR INDIAN LEGAL ACTS

: In Today’s digital age, access to legal knowledge among students is crucial for fostering a well-informed citizenry. However, comprehending the intricacies of provisions of Indian Legal Acts, especially in a complex legal system like India’s, can be daunting, even for seasoned professionals, let alone for educational purposes among children. To bridge this gap and foster legal literacy from a young age, we introduced a PDF-based Closed-domain Question-Answering (CDQA) System for Indian Legal Acts. Our system simplifies educating students about provisions of Indian Legal Acts such as RTI(Right to Education), RTE(Right to Education), Anti-Dowry Act, etc. By leveraging natural language processing(NLP) techniques and machine learning algorithms, our system, powered by LangChain, enables users to pose questions in natural language related to Indian Legal Acts. LangChain represents a groundbreaking advancement in question-answer systems, harnessing the power of cutting-edge language models to provide accurate and comprehensive responses to user queries. Developed based on state-of-the-art NLP techniques, LangChain is a versatile and highly adaptable tool that offers a versatile solution with transformative potential across diverse industries, including education, healthcare, legal research, customer support, etc.

Medical Question Summarization with Entity-driven Contrastive Learning

By summarizing longer consumer health questions into shorter and essential ones, medical question-answering systems can more accurately understand consumer intentions and retrieve suitable answers. However, medical question summarization is very challenging due to obvious distinctions in health trouble descriptions from patients and doctors. Although deep learning has been applied to successfully address the medical question summarization (MQS) task, two challenges remain: how to correctly capture question focus to model its semantic intention, and how to obtain reliable datasets to fairly evaluate performance. To address these challenges, this article proposes a novel medical question summarization framework based on e ntity-driven c ontrastive l earning (ECL). ECL employs medical entities present in frequently asked questions (FAQs) as focuses and devises an effective mechanism to generate hard negative samples. This approach compels models to focus on essential information and consequently generate more accurate question summaries. Furthermore, we have discovered that some MQS datasets, such as the iCliniq dataset with a 33% duplicate rate, have significant data leakage issues. To ensure an impartial evaluation of the related methods, this article carefully examines leaked samples to reorganize more reasonable datasets. Extensive experiments demonstrate that our ECL method outperforms the existing methods and achieves new state-of-the-art performance, i.e., 52.85, 43.16, 41.31, 43.52 in terms of ROUGE-1 metric on MeQSum, CHQ-Summ, iCliniq, HealthCareMagic dataset, respectively. The code and datasets are available at https://github.com/yrbobo/MQS-ECL

REAL-TIME SPEECH TO TEXT CONVERSION & ANSWER EVALUATION

To improve user engagement, a new question-answering system including speech recognition technology is presented in this study. Users react verbally to queries presented to them via a command-line interface built into the system. Advanced speech-to-text conversion techniques are used to convert these spoken responses into text for analysis. The system compares user responses with predefined proper responses using cosine similarity to assess input correctness. An SMTP server is used by the system to automatically send quiz scores to users via email after the quiz is finished. Showcasing the merging of spoken conversation and automated evaluation in educational environments, this ground-breaking solution offers a simplified way to interactive learning and assessment. Keywords—Question-answering, Speech recognition, Command-line interface, Speech-to-text conversion, Cosine similarity [8], User input accuracy, Quiz scores, Email notification, SMTP [4] server, Interactive learning.

GPB and BAC: two novel models towards building an intelligent motor fault maintenance question answering system

Generally, the existing methods for constructing a knowledge graph used in a question answering system adopted two different models respectively, one is for identifying entities, and the other is for extracting relationships between entities. However, this method may reduce the quality of knowledge because it is very difficult to keep contextual information consistent with the same entities in the two different models. To address this issue, this paper proposes a model called GPB (GlobalPointer + BiLSTM) which integrates the BiLSTM into GlobalPointer through concatenation operations to simultaneously guarantee the rationality of identified entities and relationships between entities. In addition, to enhance the user experience using an intelligent motor fault maintenance question answering system, a model called BAC (BiLSTM + Attention + CRF) is proposed to identify named entities in user questions, and the BERT-wwm model is used to classify user intentions to improve the quality of answers. Finally, to verify the advantages of the proposed model GPB and BAC, comparative experiments and real application effects of the developed question answering system are demonstrated on our built motor fault maintenance dataset. The experimental results indicate that the constructed knowledge graph and developed question answering system provide engineers with high-quality motor maintenance knowledge services.