Text REtrieval Conference Research Articles

A semantically enhanced text retrieval framework with abstractive summarization

AbstractRecently, large pretrained language models (PLMs) have led a revolution in the information retrieval community. In most PLMs‐based retrieval frameworks, the ranking performance broadly depends on the model structure and the semantic complexity of the input text. Sequence‐to‐sequence generative models for question answering or text generation have proven to be competitive, so we wonder whether these models can improve ranking effectiveness by enhancing input semantics. This article introduces SE‐BERT, a semantically enhanced bidirectional encoder representation from transformers (BERT) based ranking framework that captures more semantic information by modifying the input text. SE‐BERT utilizes a pretrained generative language model to summarize both sides of the candidate passage and concatenate them into a new input sequence, allowing BERT to acquire more semantic information within the constraints of the input sequence's length. Experimental results from two Text Retrieval Conference datasets demonstrate that our approach's effectiveness increasing as the length of the input text increases.

Few-shot Text Classification Method Based on Feature Optimization

For the poor effect of few-shot text classification caused by insufficient data for feature representation, this paper combines wide and deep attention bidirectional long short time memory (WDAB-LSTM) and a prototypical network to optimize text features for better classification performance. With this proposed algorithm, text enhancement and preprocessing are firstly adopted to solve the problem of insufficient samples and WDAB-LSTM is used to increase word attention to get output vectors containing important context-related information. Then the prototypical network is added to optimize the distance measurement module in the model for a better effect on feature extraction and sample representation. To test the performance of this algorithm, Amazon Review Sentiment Classification (ARSC), Text Retrieval Conference (TREC), and Kaggle are selected. Compared with the Siamese network and the prototypical network, the proposed algorithm with feature optimization has a relatively higher accuracy rate, precision rate, recall rate, and F1 value.

Relevance feedback for building pooled test collections

Offline evaluation of information retrieval systems depends on test collections. These datasets provide the researchers with a corpus of documents, topics and relevance judgements indicating which documents are relevant for each topic. Gathering the latter is costly, requiring human assessors to judge the documents. Therefore, experts usually judge only a portion of the corpus. The most common approach for selecting that subset is pooling. By intelligently choosing which documents to assess, it is possible to optimise the number of positive labels for a given budget. For this reason, much work has focused on developing techniques to better select which documents from the corpus merit human assessments. In this article, we propose using relevance feedback to prioritise the documents when building new pooled test collections. We explore several state-of-the-art statistical feedback methods for prioritising the documents the algorithm presents to the assessors. A thorough comparison on eight Text Retrieval Conference (TREC) datasets against strong baselines shows that, among other results, our proposals improve in retrieving relevant documents with lower assessment effort than other state-of-the-art adjudicating methods without harming the reliability, fairness and reusability.

ECAsT: a large dataset for conversational search and an evaluation of metric robustness.

The Text REtrieval Conference Conversational assistance track (CAsT) is an annual conversational passage retrieval challenge to create a large-scale open-domain conversational search benchmarking. However, as of yet, the datasets used are small, with just more than 1,000 turns and 100 conversation topics. In the first part of this research, we address the dataset limitation by building a much larger novel multi-turn conversation dataset for conversation search benchmarking called Expanded-CAsT (ECAsT). ECAsT is built using a multi-stage solution that uses a combination of conversational query reformulation and neural paraphrasing and also includes a new model to create multi-turn paraphrases. The meaning and diversity of paraphrases are evaluated with human and automatic evaluation. Using this methodology, we produce and release to the research community a conversational search dataset that is 665% more extensive in terms of size and language diversity than is available at the time of this study, with more than 9,200 turns. The augmented dataset not only provides more data but also more language diversity to improve conversational search neural model training and testing. In the second part of the research, we use ECAsT to assess the robustness of traditional metrics for conversational evaluation used in CAsT and identify its bias toward language diversity. Results show the benefits of adding language diversity for improving the collection of pooled passages and reducing evaluation bias. We found that introducing language diversity via paraphrases returned up to 24% new passages compared to only 2% using CAsT baseline.

PM-Search: State-of-the-art Evidence Retriever for Precision Medicine: Algorithm Development and Validation.

Under the paradigm of Precision Medicine (PM), patients with the same disease can receive different personalized therapies according to their clinical and genetic features. These therapies are determined by the totality of all available clinical evidence, including results from case reports, clinical trials, and systematic reviews. However, it is increasingly difficult for physicians to find such evidence from scientific publications, whose size is growing at an unprecedented pace. In this work, we propose the PM-Search system to facilitate the retrieval of clinical literature that contains critical evidence for or against giving specific therapies to certain cancer patients. The PM-Search system combines a Baseline Retriever that selects document candidates at a large scale and an Evidence Re-ranker that finely reorders the candidates based on their evidence quality. The Baseline Retriever uses query expansion and keyword matching with the Elasticsearch retrieval engine, and the Evidence Re-ranker fits pre-trained language models to expert annotations that are derived from an active learning strategy. The PM-Search system achieves the best performance in the retrieval of high-quality clinical evidence at the TREC PM Track 2020, outperforming the second-ranking systems by large margins (0.4780 vs. 0.4238 for standard NDCG@30 and 0.4519 vs. 0.4193 for exponential NDCG@30). We present PM-Search, a state-of-the-art search engine to assist the practicing of evidence-based PM. PM-Search uses a novel BioBERT-based active learning strategy that models evidence quality and improves the model performance. Our analyses show that evidence quality is a distinct aspect from general relevance, and specific modeling of evidence quality beyond general relevance is required for a PM search engine.

Hybrid optimization and ontology-based semantic model for efficient text-based information retrieval.

Query expansion is an important approach utilized to improve the efficiency of data retrieval tasks. Numerous works are carried out by the researchers to generate fair constructive results; however, they do not provide acceptable results for all kinds of queries particularly phrase and individual queries. The utilization of identical data sources and weighting strategies for expanding such terms are the major cause of this issue which leads the model unable to capture the comprehensive relationship between the query terms. In order to tackle this issue, we developed a novel approach for query expansion technique to analyze the different data sources namely WordNet, Wikipedia, and Text REtrieval Conference. This paper presents an Improved Aquila Optimization-based COOT(IAOCOOT) algorithm for query expansion which retrieves the semantic aspects that match the query term. The semantic heterogeneity associated with document retrieval mainly impacts the relevance matching between the query and the document. The main cause of this issue is that the similarity among the words is not evaluated correctly. To overcome this problem, we are using a Modified Needleman Wunsch algorithm algorithm to deal with the problems of uncertainty, imprecision in the information retrieval process, and semantic ambiguity of indexed terms in both the local and global perspectives. The k most similar word is determined and returned from a candidate set through the top-k words selection technique and it is widely utilized in different tasks. The proposed IAOCOOT model is evaluated using different standard Information Retrieval performance metrics to compute the validity of the proposed work by comparing it with other state-of-art techniques.

Prediction of the high-cost normalised discounted cumulative gain (nDCG) measure in information retrieval evaluation

Introduction. Information retrieval systems are vital to meeting daily information needs of users. The effectiveness of these systems has often been evaluated using the test collections approach, despite the high evaluation costs of this approach. Recent methods have been proposed that reduce evaluation costs through the prediction of information retrieval performance measures at the higher cut-off depths using other measures computed at the lower cut-off depths. The purpose of this paper is to propose two methods that addresses the challenge of accurately predicting the normalised discounted cumulative gain (nDCG) measure. Method. Data from selected test collections of the Text REtrieval Conference was used. The proposed methods employ the gradient boosting and linear regression models trained with topic scores of measures partitioned by TREC Tracks. Analysis. To evaluate the proposed methods, the coefficient of determination, Kendall's tau and Spearman correlations were used. Results. The proposed methods provide better predictions of the nDCG measure at the higher cut-off depths while using other measures computed at the lower cut-off depths. Conclusions. These proposed methods have shown improvement in the predictions of the nDCG measure while reducing the evaluation costs.

Using TREC for developing semantic information retrieval benchmark for Urdu

Information Retrieval (IR) systems are developed to fulfill several needs of users. As a plethora of IR techniques has been developed, therefore the evaluation of these techniques has been of paramount importance. The evaluation of these techniques require test collections, which are composed of a collection of documents, queries and relevance judgement between query–document pairs. Recognizing the importance of the evaluation of IR techniques, Text REtrieval Conference (TREC) has been regularly organized for the last three decades with an aim to develop and continuously improve these techniques. However, most of the resource development effort has been directed to English and other Western languages, whereas resource development for Urdu has received little attention, which has thwarted the development of IR techniques. Furthermore, the available benchmarks have several limitations which include smaller size, unavailability of the benchmark, inadequate candidate documents for evaluation, and merely binary relevance judgement. To that end, this study has focused on constructing the largest-ever and semantic IR benchmark for the Urdu language that strictly complies with the procedures proposed by TREC. That is, firstly, a large collection of 2,887,169 Urdu documents is scrapped and converted into the standard format proposed by TREC. Secondly, 105 queries are generated, which includes detailed descriptions of 35 queries and three variants of each query in the TREC format. Thirdly, a pooling-based approach is employed using queries, as well as their variants, to identify a candidate pool of 13,392 documents for human judgement. Finally, two experts performed 13,392 query–document comparisons and ranked them on a scale of four types of relevance: highly relevant, fairly relevant, marginally relevant and irrelevant, which is a significant enhancement from the existing benchmarks. The benchmark can be used for the evaluation of existing IR techniques, as well as for future techniques.

Multi-Layer Contextual Passage Term Embedding for Ad-Hoc Retrieval

Nowadays, pre-trained language models such as Bidirectional Encoder Representations from Transformer (BERT) are becoming a basic building block in Information Retrieval tasks. Nevertheless, there are several limitations when applying BERT to the query-document matching task: (1) relevance assessments are applicable at the document-level, and the tokens of documents often exceed the maximum input length of BERT; (2) applying BERT to long documents leads to a great consumption of memory usage and run time, owing to the computational cost of the interactions between tokens. This paper explores a novel multi-layer contextual passage architecture that leverage text summarization extraction to generate passage-level evidence for the pre-selected document passage thus brought new possibilities for the long document relevance task. Experiments were conducted on two standard ad-hoc retrieval collections from the Text Retrieval Conference (TREC) 2004 Robust Track (Robust04) and ClueWeb09 with two different characteristics individually. Experimental results show that our approach can significantly outperform the strong baselines and even compared with the same BERT-based models, the precision of our methods as well as state-of-the-art neural ranking models.

An Uncommon Task: Participatory Design in Legal AI

Despite growing calls for participation in AI design, there are to date few empirical studies of what these processes look like and how they can be structured for meaningful engagement with domain experts. In this paper, we examine a notable yet understudied AI design process in the legal domain that took place over a decade ago, the impact of which still informs legal automation efforts today. Specifically, we examine the design and evaluation activities that took place from 2006 to 2011 within the Text REtrieval Conference's (TREC) Legal Track, a computational research venue hosted by the National Institute of Standards and Technologies. The Legal Track of TREC is notable in the history of AI research and practice because it relied on a range of participatory approaches to facilitate the design and evaluation of new computational techniques-in this case, for automating attorney document review for civil litigation matters. Drawing on archival research and interviews with coordinators of the Legal Track of TREC, our analysis reveals how an interactive simulation methodology allowed computer scientists and lawyers to become co-designers and helped bridge the chasm between computational research and real-world, high-stakes litigation practice. In analyzing this case from the recent past, our aim is to empirically ground contemporary critiques of AI development and evaluation and the calls for greater participation as a means to address them.

A Scalable Real-Time Agent-Based Information Retrieval Engine

In distributed information retrieval systems, information in web should be ranked based on a combination of multiple features. Linear combination of ranks has been the dominant approach due to its simplicity and efficiency. Such a combination scheme in distributed infrastructure requires that ranks in resources or agents are comparable to each other. The main challenge is how to transform the raw rank values of different criteria appropriately to make them comparable before any combination. In this manuscript, we will demonstrate how to rank Web documents based on its resource-provided information stream and how to combine and incorporate several raking schemas in one time. The system was tested on the queries provided by a Text Retrieval Conference (TREC), and our experimental results showed that it is robust and efficient compared with similar platforms that used offline data resources.

A hybrid query expansion framework for the optimal retrieval of the biomedical literature

With the proliferation of biomedical literature, it is quite challenging for biomedical scientists to keep them updated with the new advancements. In biomedical literature retrieval systems, the keywords in the user-defined queries are often defined with various lexical variants consequently leading to the vocabulary mismatch (VM). One possible way to cope with these issues is to introduce a query expansion (QE) framework to enrich the original queries with the auxiliary semantically similar terms for each keyword mentioned in a query. In this research, we propose a biomedical QE framework to alleviate the VM. The proposed approach combines the clinical diagnosis information (CDI) and word embeddings (WEs) simultaneously to retrieve the relevant biomedical literature. The process of embeddings vocabulary terms as real-valued and low dimensional vectors referred to as word embedding has garnered significant attention by potentially capturing the implicit semantics. We have exploited threefold word embeddings (Domain-Specific, Domain-Agnostic, and Hybrid) and integrated the embeddings outcomes with the CDI to get the best query combination for the efficient retrieval of biomedical literature. Experimental results procured for the Text REtrieval Conference dataset showed that CDI, when used with the hybrid word embeddings surpassed the WEs trained for the domain-specific and domain-agnostic data. The results demonstrate that the utilization of this unique setup of merging two techniques is a valuable addition to the QE process leading to significantly improved precision rate and VM in biomedical literature retrieval. We hope that our approach would assist investigators to use this query combination to retrieve relevant articles.

On the TREC Blog Track

The rise of blogging as a new grassroots publishing medium and the many interesting peculiarities that characterise blogs compared to other genres of documents opened up several new interesting research areas in the information retrieval field. The Blog track was introduced in 2006 as part of the renowned Text REtrieval Conference (TREC) evaluation forum, to drive research on the blogosphere and to facilitate experimentation and evaluation of blog search techniques. This paper reports on two years of the Blog track at TREC. We describe the blog search tasks investigated at TREC, and discuss the main lessons we learnt from the track. We conclude the paper with discussions of the broader implications of the Blog track lessons and possible directions for the future, with the aim to uncover and explore the richness of information available in the blogosphere.

A New Approach of Intelligent Data Retrieval Paradigm

What is a real time agent, how does it remedy ongoing daily frustrations for users, and how does it improve the retrieval performance in World Wide Web? These are the main question we focus on this manuscript. In many distributed information retrieval systems, information in agents should be ranked based on a combination of multiple criteria. Linear combination of ranks has been the dominant approach due to its simplicity and effectiveness. Such a combination scheme in distributed infrastructure requires that the ranks in resources or agents are comparable to each other before combined. The main challenge is transforming the raw rank values of different criteria appropriately to make them comparable before any combination. Different ways for ranking agents make this strategy difficult. In this research, we will demonstrate how to rank Web documents based on resource-provided information how to combine several resources raking schemas in one time. The proposed system was implemented specifically in data provided by agents to create a comparable combination for different attributes. The proposed approach was tested on the queries provided by Text Retrieval Conference (TREC). Experimental results showed that our approach is effective and robust compared with offline search platforms.

Evaluating Real-Time Search over Tweets

Twitter offers a phenomenal platform for the social sharing of information. We describe new resources that have been created in the context of the Text Retrieval Conference (TREC) to support the academic study of Twitter as a real-time information source. We formalize an information seeking task — real-time search — and offer a methodology for measuring system effectiveness. At the TREC 2011 Microblog Track, 58 research groups participated in the first ever evaluation of this task. We present data from the effort to illustrate and support our methodology.

Building a text collection for Urdu information retrieval

Urdu is a widely spoken language in the Indian subcontinent with over 300 million speakers worldwide. However, linguistic advancements in Urdu are rare compared to those in other European and Asian languages. Therefore, by following Text Retrieval Conference standards, we attempted to construct an extensive text collection of 85 304 documents from diverse categories covering over 52 topics with relevance judgment sets at 100 pool depth. We also present several applications to demonstrate the effectiveness of our collection. Although this collection is primarily intended for text retrieval, it can also be used for named entity recognition, text summarization, and other linguistic applications with suitable modifications. Ours is the most extensive existing collection for the Urdu language, and it will be freely available for future research and academic education.

Document Retrieval for Precision Medicine Using a Deep Learning Ensemble Method.

BackgroundWith the development of biomedicine, the number of biomedical documents has increased rapidly bringing a great challenge for researchers trying to retrieve the information they need. Information retrieval aims to meet this challenge by searching relevant documents from abundant documents based on the given query. However, sometimes the relevance of search results needs to be evaluated from multiple aspects in specific retrieval tasks, thereby increasing the difficulty of biomedical information retrieval.ObjectiveThis study aimed to find a more systematic method for retrieving relevant scientific literature for a given patient.MethodsIn the initial retrieval stage, we supplemented query terms through query expansion strategies and applied query boosting to obtain an initial ranking list of relevant documents. In the re-ranking phase, we employed a text classification model and relevance matching model to evaluate documents from different dimensions and then combined the outputs through logistic regression to re-rank all the documents from the initial ranking list.ResultsThe proposed ensemble method contributed to the improvement of biomedical retrieval performance. Compared with the existing deep learning–based methods, experimental results showed that our method achieved state-of-the-art performance on the data collection provided by the Text Retrieval Conference 2019 Precision Medicine Track.ConclusionsIn this paper, we proposed a novel ensemble method based on deep learning. As shown in the experiments, the strategies we used in the initial retrieval phase such as query expansion and query boosting are effective. The application of the text classification model and relevance matching model better captured semantic context information and improved retrieval performance.

Multi-agent-based hybrid peer-to-peer system for distributed information retrieval

With the increasingly huge amount of data located in various databases and the need for users to access them, distributed information retrieval (DIR) has been at the core of the preoccupations of a number of researchers. Indeed, numerous DIR systems and architectures have been proposed including the broker-based architecture. Moreover, providing DIR with more flexibility and adaptability has led researchers thinking to build DIR with software agents. Thus, this research proposes a design and an implementation of a novel system based on the broker-based architecture and the peer-to-peer (P2P) network called broker-based P2P network. The proposed architecture is implemented with a multi-agent system (MAS) where the main agent playing the role of the broker, receives query from a peer agent and forwards them to other peer agents each with their index and resources. Upon completing retrieval process at each peer agent, results are directly sent to the peer agent that initiated the query without using the broker agent. Java Agent DEvelopment framework (JADE) is used to implement the agents and, for experiments, TERRIER (TERabyte RetRIEveR) is extended and used as the search engine to retrieve the Text Retrieval Conference (TREC) collections dataset notably TREC-6. The peer agent that originated the query progressively collects results coming from other peer agents, normalises and merges them and then proceeds with re-ranking. For normalisation, MinMax and Sum that are unsupervised normalisation methods are used.

Query expansion based on term distribution and DBpedia features

Query Expansion (QE) approaches that involve the reformulation of queries by adding new terms to the initial user query, are intended to ameliorate the vocabulary mismatch between the query keywords and the documents’ in Information Retrieval Systems (IRS). One big issue in QE is the selection of the right candidate terms for expansion. For this purpose Linked Data can be used, as a valuable resource, for providing additional expansion features such as the values of sub- and super classes of resources. The underlying research question is whether interlinked data and vocabulary items provide features which can be taken into account for query expansion. In this paper, we introduced a new QE approach that aimed at improving IRS by using the well-known distribution based method Bose-Einstein statistics (Bo1) as well as Linked Data from the knowledge base DBpedia using different numbers of expansion terms. We evaluated the effectiveness of each method individually as well as their combinations using two Text REtrieval Conference (TREC) test collections. Our approach has lead to significant improvement in terms of precision, recall, Mean Average Precision (MAP) at rank 10, and normalized Discounted Cumulative Gain (nDCG) at different ranks compared to Pseudo Relevance Feedback (PRF) that we used as a baseline. The results show that the inclusion of semantic annotations clearly improves the retrieval performance over the baseline method.

An improved BM25 algorithm for clinical decision support in Precision Medicine based on co-word analysis and Cuckoo Search

BackgroundRetrieving gene and disease information from a vast collection of biomedical abstracts to provide doctors with clinical decision support is one of the important research directions of Precision Medicine.MethodWe propose a novel article retrieval method based on expanded word and co-word analyses, also conducting Cuckoo Search to optimize parameters of the retrieval function. The main goal is to retrieve the abstracts of biomedical articles that refer to treatments. The methods mentioned in this manuscript adopt the BM25 algorithm to calculate the score of abstracts. We, however, propose an improved version of BM25 that computes the scores of expanded words and co-word leading to a composite retrieval function, which is then optimized using the Cuckoo Search. The proposed method aims to find both disease and gene information in the abstract of the same biomedical article. This is to achieve higher relevance and hence score of articles. Besides, we investigate the influence of different parameters on the retrieval algorithm and summarize how they meet various retrieval needs.ResultsThe data used in this manuscript is sourced from medical articles presented in Text Retrieval Conference (TREC): Clinical Decision Support (CDS) Tracks of 2017, 2018, and 2019 in Precision Medicine. A total of 120 topics are tested. Three indicators are employed for the comparison of utilized methods, which are selected among the ones based only on the BM25 algorithm and its improved version to conduct comparable experiments. The results showed that the proposed algorithm achieves better results.ConclusionThe proposed method, an improved version of the BM25 algorithm, utilizes both co-word implementation and Cuckoo Search, which has been verified achieving better results on a large number of experimental sets. Besides, a relatively simple query expansion method is implemented in this manuscript. Future research will focus on ontology and semantic networks to expand the query vocabulary.