End-to-End Contextualized Document Indexing and Retrieval with Neural Networks

Abstract

Relevance in ad-hoc retrieval is a fundamental problem of text understanding. Developing neural network methods for this foundational task of Information Retrieval (IR) has the potential to impact many search domains. Recently, a new generation of Transformerbased [6] neural re-ranking models initiated a new era, by providing substantial effectiveness increases in ad-hoc search tasks [1, 4, 5]. They operate on the full text of a query and a list of candidate documents from an initial retrieval. Using self-attention, they contextualize term occurrences conditioned on the sequence they are contained in. However, self-attention, because it is applied to a whole sequence and commonly uses many layers, is inefficient and re-ranking models still depended on the bottleneck of the initial retrieval of candidate documents. In our thesis we plan to address these shortcomings. First, we plan to make contextualization efficient enough to to be usable in resource constrained environments, and second we plan to use the efficient contextualization components to create a novel approach for learning to index and retrieve in a unified neural model. Hence, our main research questions are: RQ1 How can we balance the trade-off between efficiency and effectiveness in contextualized neural re-ranking? Solving efficiency problems with massively parallelized hardware is neither an economically nor environmentally friendly solution. We wish to tackle the problem of efficient contextualization in two common scenarios of ad-hoc ranking: passage retrieval and document retrieval. As part of this PhD we already proposed a novel efficient Transformer based passage re-ranking model: The TK (Transformer-Kernel) model [4] utilizes shallow Transformerlayers to contextualize query and passages separately, and the kernel-pooling technique [7] to score individual term interactions between a query and a passage. Additionally, we explored localattention as an effective approach for document-length ranking with an extension to the TK for long text (TKL) [3]. RQ2 What is needed to learn generalized contextualized representations for indexing and retrieval? Currently, IR systems are a patchwork of traditional and neural systems. Different parts of the pipeline are optimized in isolation and missing integration during training. To overcome bottlenecks and sub-optimal integration of pipeline components, we plan to develop a unified neural index and ranking model. A unified indexing and ranking model needs to learn how to efficiently store document representations, for example via trained sparsity [8], followed by the integration of all second-generation contextualized re-ranking components. We plan to train this unified model in a true end-toend approach, with gradients flowing through the complete system. The main challenge here is to generalize models beyond their training domain, to match the usability of traditional relevance models such as BM25, which can be used as a drop-in approach in almost any domain and language. As part of this PhD we already showed the importance of tuning the re-ranking depth as the interface between traditional and neural models [2]. Furthermore, we plan to create a truly novel indexing and ranking model, that removes all bottlenecks and patchwork systems. Contextualization is a key element to focus the model on the actual topic of a query and document, and reduce the search space for previously ambiguous words and topics. Parallel to our main research questions, we plan to keep explainability and thorough analysis of our approaches a priority during this PhD work.