Natural Language Information Retrieval Research Articles

Semantic Verification in Large Language Model-based Retrieval Augmented Generation

This position paper presents a novel approach of semantic verification in Large Language Model-based Retrieval Augmented Generation (LLM-RAG) systems, focusing on the critical need for factually accurate information dissemination during public debates, especially prior to plebiscites e.g. in direct democracies, particularly in the context of Switzerland. Recognizing the unique challenges posed by the current generation of Large Language Models (LLMs) in maintaining factual integrity, this research proposes an innovative solution that integrates retrieval mechanisms with enhanced semantic verification processes. The paper outlines a comprehensive methodology following a Design Science Research approach, which includes defining user personas, designing conversational interfaces, and iteratively developing a hybrid dialogue system. Central to this system is a robust semantic verification framework that leverages a knowledge graph for fact-checking and validation, ensuring the correctness and consistency of information generated by LLMs. The paper discusses the significance of this research in the context of Swiss direct democracy, where informed decision-making is pivotal. By improving the accuracy and reliability of information provided to the public, the proposed system aims to support the democratic process, enabling citizens to make well-informed decisions on complex issues. The research contributes to advancing the field of natural language processing and information retrieval, demonstrating the potential of AI and LLMs in enhancing civic engagement and democratic participation.

Research Paper Summarization and Recommendation

The exponential growth of academic literature has made it increasingly challenging for researchers to keep up with the latest developments in their respective fields. This paper presents an approach for summarizing and recommending research papers to help researchers efficiently manage and navigate the vast amount of academic literature. We propose using a transformer-based model for generating abstractive summaries of research papers and a cosine similarity-based recommendation system for suggesting similar papers to a given paper. The proposed approach is evaluated on a dataset of research papers, and the results show that it is effective in generating coherent and concise summaries and recommending relevant research papers. The findings of this study have important implications for researchers and practitioners in the field of natural language processing and information retrieval.

“An Extensive study of Symantic and Syntatic Approaches to Automatic Text Summarization”

Automatic text summarization (ATS) has emerged as a crucial research domain in the discipline of natural language processing (NLP) and information retrieval. The exponential growth of digital content has necessitated the need for efficient techniques that can automatically generate concise and informative summaries from lengthy documents. This article provided a comprehensive recap of automatic text summarization, covering both abstractive and extractive methods. Using extractive techniques, prime phrases or keywords from the original text are identified and chosen, while abstractive methods involve producing summaries by paraphrasing and synthesizing content in a more human-like manner. Discussed the advantages and limitations of each approach, including the challenge of ATS, which arises when summarizing content from external sources. Furthermore, reviews common evaluation metrics used for assessing the quality of summaries and discusses recent advancements in neural network-based approaches for text summarization. This survey aims to provide an overview of automatic text summarization which acts as a useful resource for researchers and practitioners in the fields of information retrieval and NLP.

Enriching Portuguese Medieval Texts with Named Entity Recognition

Historical data poses unique challenges to natural language processing (NLP) and information retrieval (IR) tools, including digitization errors, lack of annotated data, and diachronic-specific issues. However, the increasing recognition of the value in historical documents has promoted efforts to semantically enrich and optimize their analysis. This article contributes to this endeavour by enriching the Corpus de Textos Antigos through NLP tools and techniques to enhance its usability and support research. The corpus undergoes linguistic annotation, including part-of-speech tagging, lemma annotation and named entity recognition (NER). Subsequently, the article delves into the tasks of entity disambiguation and entity linking, which involve identifying and disambiguating named entities by referring to a knowledge base (KB). Addressing the challenges posed by factors such as text state, epoch and the chosen KB, the article presents insights into related work, annotation results and the linguistic interest of a medieval annotated corpus for named entities. It concludes by discussing the challenges and providing avenues for future research in this domain.

Quranic Studies Evolution: A Bibliometric Analysis from 1880 to 2023

This analysis focuses on the study of the Quran, from 1880 to 2023 using data obtained from Scopus. Made accessible through a BibTeX file on GitHub. The examination covers aspects of research trends including the number of citations per year the influence of sources notable authors and their affiliations the countries of corresponding authors the production trends in different countries over time and a network illustrating how key concepts are interconnected. By conducting an analysis several significant findings were discovered. The data demonstrated an increase in citations per year indicating a growing importance and scholarly interest in Quranic Studies. Noteworthy authors such as Rippin A and Reynolds GS as prominent affiliations like the University of Malaya and International Islamic University Malaysia made substantial contributions to this field. The distribution of corresponding authors countries highlighted Malaysias, Indonesias and Irans roles. Furthermore when examining production trends over time for countries it became evident that contributions from the United Kingdom, USA,Malaysia,Iran and Indonesia had seen growth. Additionally the interconnectedness between concepts such as natural language processing systems, computational linguistics,semantics and information retrieval were emphasized in the cooccurrence network, within Quranic Studies. This, in depth analysis provides insights into the development significant contributors and intersections of themes in the study of the Quran. It establishes a foundation for research endeavors, in this area.

Knowledge-based Visual Question Answering about Named Entities

This thesis is positioned at the intersection of several research fields, Natural Language Processing, Information Retrieval (IR) and Computer Vision, which have unified around representation learning and pre-training methods. We have defined and studied a new multimodal task: Knowledge-based Visual Question Answering about Named Entities (KVQAE). We were particularly interested in cross-modal interactions and different ways of representing named entities. We also focused on data used to train and, more importantly, evaluate Question Answering systems through different metrics. We annotated a dataset for this purpose, the first in KVQAE comprising various types of entities. We also defined an experimental framework for dealing with KVQAE in two stages through an unstructured knowledge base and identified IR as the main bottleneck of KVQAE, especially for questions about non-person entities. To improve the IR stage, we studied different multimodal fusion methods, which are pre-trained through an original task: the Multimodal Inverse Cloze Task. We found that these models leveraged a cross-modal interaction that we had not originally considered, and which may address the heterogeneity of visual representations of named entities. These results were strengthened by a study of the CLIP model, which allows this cross-modal interaction to be modeled directly. Awarded by : Université Paris-Saclay, Orsay, France on 8 November 2023. Supervised by : Olivier Ferret and Camille Guinaudeau. Available at : https://www.theses.fr/s247993.

Automatic Context Pattern Generation for Entity Set Expansion

Entity Set Expansion (ESE) is a valuable task that aims to find entities of the target semantic class described by given seed entities. Various Natural Language Processing (NLP) and Information Retrieval (IR) downstream applications have benefited from ESE due to its ability to discover knowledge. Although existing corpus-based ESE methods have achieved great progress, they still rely on corpora with high-quality entity information annotated, because most of them need to obtain the context patterns through the position of the entity in a sentence. Therefore, the quality of the given corpora and their entity annotation has become the bottleneck that limits the performance of such methods. To overcome this dilemma and make the ESE models free from the dependence on entity annotation, our work aims to explore a new ESE paradigm, namely corpus-independent ESE. Specifically, we devise a context pattern generation module that utilizes autoregressive language models (e.g., GPT-2) to automatically generate high-quality context patterns for entities. In addition, we propose the GAPA, a novel ESE framework that leverages the aforementioned <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b> ener <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> ted <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PA</b> tterns to expand target entities. Extensive experiments and detailed analyses on three widely used datasets demonstrate the effectiveness of our method. All the codes of our experiments are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/geekjuruo/GAPA</uri> .

Automatic genre identification: a survey

AbstractAutomatic genre identification (AGI) is a text classification task focused on genres, i.e., text categories defined by the author’s purpose, common function of the text, and the text’s conventional form. Obtaining genre information has been shown to be beneficial for a wide range of disciplines, including linguistics, corpus linguistics, computational linguistics, natural language processing, information retrieval and information security. Consequently, in the past 20 years, numerous researchers have collected genre datasets with the aim to develop an efficient genre classifier. However, their approaches to the definition of genre schemata, data collection and manual annotation vary substantially, resulting in significantly different datasets. As most AGI experiments are dataset-dependent, a sufficient understanding of the differences between the available genre datasets is of great importance for the researchers venturing into this area. In this paper, we present a detailed overview of different approaches to each of the steps of the AGI task, from the definition of the genre concept and the genre schema, to the dataset collection and annotation methods, and, finally, to machine learning strategies. Special focus is dedicated to the description of the most relevant genre schemata and datasets, and details on the availability of all of the datasets are provided. In addition, the paper presents the recent advances in machine learning approaches to automatic genre identification, and concludes with proposing the directions towards developing a stable multilingual genre classifier.

Large Language Models and Information Retrieval

This research article explores the synergistic integration of Optical Character Recognition (OCR) technology and Large Language Models (LLMs) to advance Information Retrieval (IR) processes. In a data-centric society, efficient IR is imperative, and the combination of OCR and LLMs presents a powerful solution. OCR transforms diverse document types into machine-readable formats, while LLMs excel in language understanding and generation. The article delves into the technical intricacies of these technologies, their seamless integration, and their potential to revolutionize information retrieval. By investigating their collaborative capabilities, this research contributes to the evolving landscape of natural language processing and information retrieval systems.

A hybrid part-of-speech tagger with annotated Kurdish corpus: advancements in POS tagging

Abstract With the rapid growth of online content written in the Kurdish language, there is an increasing need to make it machine-readable and processable. Part of speech (POS) tagging is a critical aspect of natural language processing (NLP), playing a significant role in applications such as speech recognition, natural language parsing, information retrieval, and multiword term extraction. This study details the creation of the DASTAN corpus, the first POS-annotated corpus for the Sorani Kurdish dialect. The corpus, containing 74,258 words and thirty-eight tags, employs a hybrid approach utilizing the bigram hidden Markov model in combination with the Kurdish rule-based approach to POS tagging. This approach addresses two key problems that arise with rule-based approaches, namely misclassified words and ambiguity-related unanalyzed words. The proposed approach’s accuracy was assessed by training and testing it on the DASTAN corpus, yielding a 96% accuracy rate. Overall, this study’s findings demonstrate the effectiveness of the proposed hybrid approach and its potential to enhance NLP applications for Sorani Kurdish.

Systematic Literature Review on Ontology-based Indonesian Question Answering System

Question-Answering (QA) systems at the intersection of natural language processing, information retrieval, and knowledge representation aim to provide efficient responses to natural language queries. These systems have seen extensive development in English and languages like Indonesian present unique challenges and opportunities. This literature review paper delves into the state of ontology-based Indonesian QA systems, highlighting critical challenges. The first challenge lies in sentence understanding, variations, and complexity. Most systems rely on syntactic analysis and struggle to grasp sentence semantics. Complex sentences, especially in Indonesian, pose difficulties in parsing, semantic interpretation, and knowledge extraction. Addressing these linguistic intricacies is pivotal for accurate responses. Secondly, template-based SPARQL query construction, commonly used in Indonesian QA systems, suffers from semantic gaps and inflexibility. Advanced techniques like semantic matching algorithms and dynamic template generation can bridge these gaps and adapt to evolving ontologies. Thirdly, lexical gaps and ambiguity hinder QA systems. Bridging vocabulary mismatches between user queries and ontology labels remains a challenge. Strategies like synonym expansion, word embedding, and ontology enrichment must be explored further to overcome these challenges. Lastly, the review discusses the potential of developing multi-domain ontologies to broaden the knowledge coverage of QA systems. While this presents complex linguistic and ontological challenges, it offers the advantage of responding to various user queries across various domains. This literature review identifies crucial challenges in developing ontology-based Indonesian QA systems and suggests innovative approaches to address these challenges.

TA-DA: Topic-Aware Domain Adaptation for Scientific Keyphrase Identification and Classification (Student Abstract)

Keyphrase identification and classification is a Natural Language Processing and Information Retrieval task that involves extracting relevant groups of words from a given text related to the main topic. In this work, we focus on extracting keyphrases from scientific documents. We introduce TA-DA, a Topic-Aware Domain Adaptation framework for keyphrase extraction that integrates Multi-Task Learning with Adversarial Training and Domain Adaptation. Our approach improves performance over baseline models by up to 5% in the exact match of the F1-score.

Semantic Tagging for the Urdu Language: Annotated Corpus and Multi-Target Classification Methods

Extracting and analysing meaning-related information from natural language data has attracted the attention of researchers in various fields, such as natural language processing, corpus linguistics, information retrieval, and data science. An important aspect of such automatic information extraction and analysis is the annotation of language data using semantic tagging tools. Different semantic tagging tools have been designed to carry out various levels of semantic analysis, for instance, named entity recognition and disambiguation, sentiment analysis, word sense disambiguation, content analysis, and semantic role labelling. Common to all of these tasks, in the supervised setting, is the requirement for a manually semantically annotated corpus, which acts as a knowledge base from which to train and test potential word and phrase-level sense annotations. Many benchmark corpora have been developed for various semantic tagging tasks, but most are for English and other European languages. There is a dearth of semantically annotated corpora for the Urdu language, which is widely spoken and used around the world. To fill this gap, this study presents a large benchmark corpus and methods for the semantic tagging task for the Urdu language. The proposed corpus contains 8,000 tokens in the following domains or genres: news, social media, Wikipedia, and historical text (each domain having 2K tokens). The corpus has been manually annotated with 21 major semantic fields and 232 sub-fields with the USAS (UCREL Semantic Analysis System) semantic taxonomy which provides a comprehensive set of semantic fields for coarse-grained annotation. Each word in our proposed corpus has been annotated with at least one and up to nine semantic field tags to provide a detailed semantic analysis of the language data, which allowed us to treat the problem of semantic tagging as a supervised multi-target classification task. To demonstrate how our proposed corpus can be used for the development and evaluation of Urdu semantic tagging methods, we extracted local, topical and semantic features from the proposed corpus and applied seven different supervised multi-target classifiers to them. Results show an accuracy of 94% on our proposed corpus which is free and publicly available to download.

The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval

On a wide range of natural language processing and information retrieval tasks, transformer-based models, particularly pre-trained language models like BERT, have demonstrated tremendous effectiveness. Due to the quadratic complexity of the self-attention mechanism, however, such models have difficulties processing long documents. Recent works dealing with this issue include truncating long documents, in which case one loses potential relevant information, segmenting them into several passages, which may lead to miss some information and high computational complexity when the number of passages is large, or modifying the self-attention mechanism to make it sparser as in sparse-attention models, at the risk again of missing some information. We follow here a slightly different approach in which one first selects key blocks of a long document by local query-block pre-ranking, and then few blocks are aggregated to form a short document that can be processed by a model such as BERT. Experiments conducted on standard Information Retrieval datasets demonstrate the effectiveness of the proposed approach.

Hybrid support vector machine and K‐nearest neighbor‐based software testing for educational assistant

AbstractIn terms of training students for work in diverse firms, traditional and out‐of‐date teaching techniques cannot compete with digital teaching methods. To overcome this problem, the teaching approach and content must be changed. An Educational Assistant for Software Testing (EAST) framework is developed in this work to train students to improve their skills in software testing via Computer Assisted Instruction (CAI) built using Natural Language Processing (NLP), Machine learning, and information retrieval techniques. In this paper, a Group Search Optimized two‐stage hybrid Support Vector Machine‐K‐Nearest Neighbor (SVM‐KNN) classifier is used to develop a novel approach for analyzing the parameters that introduce bugs in bug reports. To decrease the data sparsity problem, the group search optimization (GSO) algorithm is used to improve the parameter selection process of the two‐stage hybrid classifier by generating optimal values for parameters such as k, c, and gamma. Two bug report datasets were used to test the model. The database for our application is built by collecting bug reports from a wide open‐source community as well as several mobile application development companies. Based on the extensive experiments conducted via different performance metrics, we can conclude that the EAST framework can improve outdated teaching methodologies.

A Survey of Information Extraction Based on Deep Learning

As a core task and an important link in the fields of natural language understanding and information retrieval, information extraction (IE) can structure and semanticize unstructured multi-modal information. In recent years, deep learning (DL) has attracted considerable research attention to IE tasks. Deep learning-based entity relation extraction techniques have gradually surpassed traditional feature- and kernel-function-based methods in terms of the depth of feature extraction and model accuracy. In this paper, we explain the basic concepts of IE and DL, primarily expounding on the research progress and achievements of DL technologies in the field of IE. At the level of IE tasks, it is expounded from entity relationship extraction, event extraction, and multi-modal information extraction three aspects, and creates a comparative analysis of various extraction techniques. We also summarize the prospects and development trends in DL in the field of IE as well as difficulties requiring further study. It is believed that research can be carried out in the direction of multi-model and multi-task joint extraction, information extraction based on knowledge enhancement, and information fusion based on multi-modal at the method level. At the model level, further research should be carried out in the aspects of strengthening theoretical research, model lightweight, and improving model generalization ability.

Twitter Big Data as a Resource for Exoskeleton Research: A Large-Scale Dataset of about 140,000 Tweets from 2017–2022 and 100 Research Questions

The exoskeleton technology has been rapidly advancing in the recent past due to its multitude of applications and diverse use cases in assisted living, military, healthcare, firefighting, and industry 4.0. The exoskeleton market is projected to increase by multiple times its current value within the next two years. Therefore, it is crucial to study the degree and trends of user interest, views, opinions, perspectives, attitudes, acceptance, feedback, engagement, buying behavior, and satisfaction, towards exoskeletons, for which the availability of Big Data of conversations about exoskeletons is necessary. The Internet of Everything style of today’s living, characterized by people spending more time on the internet than ever before, with a specific focus on social media platforms, holds the potential for the development of such a dataset by the mining of relevant social media conversations. Twitter, one such social media platform, is highly popular amongst all age groups, where the topics found in the conversation paradigms include emerging technologies such as exoskeletons. To address this research challenge, this work makes two scientific contributions to this field. First, it presents an open-access dataset of about 140,000 Tweets about exoskeletons that were posted in a 5-year period from 21 May 2017 to 21 May 2022. Second, based on a comprehensive review of the recent works in the fields of Big Data, Natural Language Processing, Information Retrieval, Data Mining, Pattern Recognition, and Artificial Intelligence that may be applied to relevant Twitter data for advancing research, innovation, and discovery in the field of exoskeleton research, a total of 100 Research Questions are presented for researchers to study, analyze, evaluate, ideate, and investigate based on this dataset.

Detecting non-natural language artifacts for de-noising bug reports

Textual documents produced in the software engineering process are a popular target for natural language processing (NLP) and information retrieval (IR) approaches. However, issue tickets often contain artifacts such as code snippets, log outputs and stack traces. These artifacts not only inflate the issue ticket sizes, but also can this noise constitute a real problem for some NLP approaches, and therefore has to be removed in the pre-processing of some approaches. In this paper, we present a machine learning based approach to classify textual content into natural language and non-natural language artifacts at line level. We show how data from GitHub issue trackers can be used for automated training set generation, and present a custom preprocessing approach for the task of artifact removal. The training sets are automatically created from Markdown annotated issue tickets and project documentation files. We use these generated training sets to train a Markdown agnostic model that is able to classify un-annotated content. We evaluate our approach on issue tickets from projects written in C++, Java, JavaScript, PHP, and Python. Our approach achieves ROC-AUC scores between 0.92 and 0.96 for language-specific models. A multi-language model trained on the issue tickets of all languages achieves ROC-AUC scores between 0.92 and 0.95. The provided models are intended to be used as noise reduction pre-processing steps for NLP and IR approaches working on issue tickets.

SBugLocater: Bug Localization Based on Deep Matching and Information Retrieval

Bug localization is a technology that locates buggy source files using bug reports reported by users. Automatic localization of buggy files can speed up the process of bug fixing to improve the efficiency and productivity of software quality assurance teams. Nowadays, some research studies have investigated the natural language information retrieval technology, but few of them have applied the matching technology in deep learning to bug localization. Therefore, we propose a bug localization model SBugLocater based on deep matching and IR. The model composes of three layers: semantic matching layer, relevance matching layer, and IR layer. In particular, the relevance matching layer captures fine-grained local matching signals, while coarse-grained semantic similarity signals come from the semantic matching layer. Further, based on collaborative filtering in different directions, the IR layer works to find whether bug reports and source files are related, which indirectly transforms the matching task of different grammatical structures between bug reports and source files into the same structure and solves the mismatching problem of the first two matching models when the query is short. In our work, four benchmark data sets are used as experimental data sets and Accuracy@k, MAP, and MRR as evaluation metrics, which are used to compare and analyze the performance of bug localization with the four state-of-the-art methods. Experimental results show that SBugLocater outperforms the four models. For example, compared with the best of the four models, the evaluation metrics of Accuracy@10, MAP, and MRR are improved by 6.9%, 13.9%, and 17%, respectively.

N-Gram Feature for Comparison of Machine Learning Methods on Sentiment in Financial News Headlines

&#x0D; &#x0D; &#x0D; Sentiment analysis is currently widely used in natural language processing or information retrieval applications. Sentiment analysis analysis can provide information related to outstanding financial news headlines and provide input to the company. Positive sentiment will also have a good impact on the development of the company, but negative sentiment will damage the company's reputation. This will affect the company's development. This study compares machine learning methods on financial news headlines with n-gram feature extraction. The purpose of this study was to obtain the best method for classifying the headline sentiment of the company's financial news. The machine learning methods compared are Multinomial Naïve Bayes, Logistic Regression, Support Vector Machine, multi-layer perceptron (MLP), Stochastic Gradient Descent, and Decision Trees. The results show that the best method is logistic regression with a percentage of f1-measure, precision, and recal of 73.94 %, 73.94 %, and 74.63 %. This shows that the n-gram and machine learning features have successfully carried out sentiment analysis. &#x0D; &#x0D; &#x0D;