IR Tasks Research Articles

Report on the 1st Workshop on Implicit Author Characterization from Texts for Search and Retrieval (IACT 2023) at SIGIR 2023

The first edition of the International Workshop on Implicit Author Characterization from Texts for Search and Retrieval (IACT'23) was held on July 27 th , 2023, in conjunction with the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) in Taipei, Taiwan. To support both online and in-person participation, the workshop was held as a hybrid event. Online participation was allowed using the EventX platform. During the course of the day, about 15 attendees had the opportunity to follow up and discuss the recent advances in topics related to identifying and extracting implicit information about authors (e.g., human or AI) from texts and using it in IR tasks. The workshop program included two invited keynotes, four paper presentations, and the report of the ColiE competition which was held as a part of the workshop1. The proceedings of the workshop are available online at https://ceur-ws.org/Vol-3477/. Date : 27 July 2023. Website : https://en.sce.ac.il/news/iact23.

NIR-Prompt: A Multi-task Generalized Neural Information Retrieval Training Framework

Information retrieval aims to find information that meets users’ needs from the corpus. Different needs correspond to different IR tasks such as document retrieval, open-domain question answering, retrieval-based dialogue, and so on, while they share the same schema to estimate the relationship between texts. It indicates that a good IR model can generalize to different tasks and domains. However, previous studies indicate that state-of-the-art neural information retrieval (NIR) models, e.g., pre-trained language models (PLMs) are hard to generalize. It is mainly because the end-to-end fine-tuning paradigm makes the model overemphasize task-specific signals and domain biases but loses the ability to capture generalized essential signals. To address this problem, we propose a novel NIR training framework named NIR-Prompt for retrieval and reranking stages based on the idea of decoupling signal capturing and combination. NIR-Prompt exploits Essential Matching Module (EMM) to capture the essential matching signals and gets the description of tasks by Matching Description Module (MDM). The description is used as task-adaptation information to combine the essential matching signals to adapt to different tasks. Experiments under in-domain multi-task, out-of-domain multi-task, and new task adaptation settings show that NIR-Prompt can improve the generalization of PLMs in NIR for both retrieval and reranking stages compared with baselines.

MedCPT: Contrastive Pre-trained Transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval.

Information retrieval (IR) is essential in biomedical knowledge acquisition and clinical decision support. While recent progress has shown that language model encoders perform better semantic retrieval, training such models requires abundant query-article annotations that are difficult to obtain in biomedicine. As a result, most biomedical IR systems only conduct lexical matching. In response, we introduce MedCPT, a first-of-its-kind Contrastively Pre-trained Transformer model for zero-shot semantic IR in biomedicine. To train MedCPT, we collected an unprecedented scale of 255 million user click logs from PubMed. With such data, we use contrastive learning to train a pair of closely integrated retriever and re-ranker. Experimental results show that MedCPT sets new state-of-the-art performance on six biomedical IR tasks, outperforming various baselines including much larger models, such as GPT-3-sized cpt-text-XL. In addition, MedCPT also generates better biomedical article and sentence representations for semantic evaluations. As such, MedCPT can be readily applied to various real-world biomedical IR tasks. The MedCPT code and model are available at https://github.com/ncbi/MedCPT.

Analysis on Content Based Image Retrieval Using Image Enhancement and Deep Learning Convolutional Neural Networks

"Content-Based" means that an image contents search analyzes instead of meta data, including keywords, tags or image descriptions. The word contents could apply in this sense to colours, structures, textures or any details extracted from the picture itself. CBIR is desirable as searches relying purely on metadata depend on the quality and the completeness of annotations. CBIR method for the recovery of images from huge, unshaped image databases is commonly used. The CBIR method is used. Therefore, users are not satisfied with standard knowledge collection methods. In addition, there are more images available to users, as well as the advent of web creation and transmission networks. Consequently, there is a permanent and important output of digital images in many regions. Hence the rapid access to these enormous picture collections and the identical image retrieval from this broad image collection presents major challenges and demands efficient techniques. The efficiency of a content-based image retrieval system depends on the characteristic representation and similarity calculation. We therefore have a simple but powerful, profound, CNN-based, and feature-extraction and classification-based imaging system. Some promising results have been obtained from a range of empirical studies on a variety of CB IR tasks through the image database. Content-based image recovery systems (CBIR) allow you to find images identical to a query image among a picture dataset. The best-known CBIR system is Google's search by image feature.

Neural Network Towards Uncertain Legal Case Retrieval

Information retrieval in natural language processing is nothing but the process of finding information that matches or satisfies information needed from a large database. Legal Case Retrieval (LCR) is considered to be an uncertain task focused to retrieve a similar case given a query case. Unlike other IR tasks, LCR is more thought-provoking since the query case is much longer and multifaceted when compared to queries in other tasks. Other than that, meaning of similarity between the query case and candidate cases is beyond the topical relevance and hence difficult to build a large-scale case retrieval dataset. Hence, this paper proposes a neural network-based retrieval model that identifies the most similar case supporting the query case decision. To achieve a high recall value, the retrieval process is pipelined into two stages, (i) the candidate cases are initially identified with matching section codes from the query case. (ii) Capture semantic relationship at the point level and then calculate the similarity between query case and candidate case by aggregating point level similarity. With the extensive experiments conducted on the benchmark LCR, the proposed model outperforms the existing work with the value of F-measure 0.82. The initial step of choosing the candidate cases with the matching section code yields a higher recall value and in turn higher F-measure.

A Unified Pretraining Framework for Passage Ranking and Expansion

Pretrained language models have recently advanced a wide range of natural language processing tasks. Nowadays, the application of pretrained language models to IR tasks has also achieved impressive results. Typical methods either directly apply a pretrained model to improve the re-ranking stage, or use it to conduct passage expansion and term weighting for first-stage retrieval. We observe that the passage ranking and passage expansion tasks share certain inherent relations, and can benefit from each other. Therefore, in this paper, we propose a general pretraining framework to enhance both tasks with Unified Encoder-Decoder networks (UED). The overall ranking framework consists of two parts in a cascade manner: (1) passage expansion with a pretraining-based query generation method; (2) re-ranking of passage candidates from a traditional retrieval method with a pretrained transformer encoder. Both the two parts are based on the same pretrained UED model, where we jointly train the passage ranking and query generation tasks for further improving the full ranking pipeline. An extensive set of experiments have been conducted on two large-scale passage retrieval datasets to demonstrate the state-of-the-art results of the proposed framework in both the first-stage retrieval and the final re-ranking. In addition, we successfully deploy the framework to our online production system, which can stably serve industrial applications with a request volume of up to 100 QPS in less than 300ms.

Bringing semantics into word image representation

The shift from one-hot to distributed representation, popularly referred to as word embedding has changed the landscape of natural language processing (nlp) and information retrieval (ir) communities. In the domain of document images, we have always appreciated the need for learning a holistic word image representation which is popularly used for the task of word spotting. The representations proposed for word spotting is different from word embedding in text since the later captures the semantic aspects of the word which is a crucial ingredient to numerous nlp and ir tasks. In this work, we attempt to encode the notion of semantics into word image representation by bringing the advancements from the textual domain. We propose two novel forms of representations where the first form is designed to be inflection invariant by focusing on the approximate linguistic root of the word, while the second form is built along the lines of recent textual word embedding techniques such as Word2Vec. We observe that such representations are useful for both traditional word spotting and also enrich the search results by accounting the semantic nature of the task. We conduct our experiments on the challenging document images taken from historical-modern collections, handwritten-printed domains, and Latin-Indic scripts. For the purpose of semantic evaluation, we have prepared a large synthetic word image dataset and report interesting results for the standard semantic evaluation metrics such as word analogy and word similarity.

Identifying Repeated Sections within Documents

Identifying sections containing a logically coherent text about a particular aspect is important for fine-grained IR, question-answering and information extraction. We propose a novel problem of identifying repeated sections, such as project details in resumes and different sports events in the transcript of a news broadcast. We focus on resumes and present four techniques (2 unsupervised, 2 supervised) for automatically identifying repeated project sections. The knowledge-based method is modeled after the human way closely. The other methods are based on integer linear programming and sequence labeling. The proposed techniques are general and can be used for identifying other kinds of repeated sections (and even non-repeating sections) in different types of documents. We compared the four methods on a dataset of resumes of IT professionals and also evaluated the benefits of identifying such repeated sections in practical IR tasks. To the best of our knowledge, this paper is the first to propose and solve the problem of repeated sections identification.

Offline versus Online Representation Learning of Documents Using External Knowledge

An intensive recent research work investigated the combined use of hand-curated knowledge resources and corpus-driven resources to learn effective text representations. The overall learning process could be run by online revising the learning objective or by offline refining an original learned representation. The differentiated impact of each of the learning approaches on the quality of the learned representations has not been studied so far in the literature. This article focuses on the design of comparable offline vs. online knowledge-enhanced document representation learning models and the comparison of their effectiveness using a set of standard IR and NLP downstream tasks. The results of quantitative and qualitative analyses show that (1) offline vs. online learning approaches have dissimilar result trends regarding the task as well as the dataset distribution counts with regard to domain application; (2) while considering external knowledge resources is undoubtedly beneficial, the way used to express relational constraints could affect semantic inference effectiveness. The findings of this work present opportunities for the design of future representation learning models, but also for providing insights about the evaluation of such models.

Multi-Noise and Multi-Channel Derived Prior Information for Grayscale Image Restoration

Image restoration is an extensively studied area with lots of outstanding algorithms developed. Nevertheless, most existing methods still have some limitations that only apply to a single tailored restoration task or suffer from long iterative reconstruction time or yield unstable results. To address these challenges, this work presents a multi-noise and multi-channel enhanced Deep Mean-Shift Prior (MEDMSP) for grayscale IR tasks. Specifically, we draw valuable high-dimensional prior knowledge by learning a multi-noise stimulated DMSP network from color images with RGB-channels. Variable augmentation technique is then adopted for incorporating the higher-dimensional network prior into the iterative reconstruction procedure. MEDMSP has been evaluated on different IR tasks and compared to a variety of state-of-the-art methods. Experimental results show that the proposed method has better capability in image deblurring and accurate compressive sensing reconstructions in terms of both visual and quantitative comparisons.

From Question to Text

Understanding unstructured texts is an essential skill for human beings as it enables knowledge acquisition. Although understanding unstructured texts is easy for we human beings with good education, it is a great challenge for machines. Recently, with the rapid development of artificial intelligence techniques, researchers put efforts to teach machines to understand texts and justify the educated machines by letting them solve the questions upon the given unstructured texts, inspired by the reading comprehension test as we humans do. However, feature effectiveness with respect to different questions significantly hinders the performance of answer selection, because different questions may focus on various aspects of the given text and answer candidates. To solve this problem, we propose a question-oriented feature attention (QFA) mechanism, which learns to weight different engineering features according to the given question, so that important features with respect to the specific question is emphasized accordingly. Experiments on MCTest dataset have well-validated the effectiveness of the proposed method. Additionally, the proposed QFA is applicable to various IR tasks, such as question answering and answer selection. We have verified the applicability on a crawled community-based question-answering dataset.

Report on the Second SIGIR Workshop on Neural Information Retrieval (Neu-IR'17)

The second SIGIR workshop on neural information retrieval (Neu-IR?17) took place on August 11, 2017, in Tokyo, Japan. Following the successful 2016 edition, the workshop continued to serve as a forum for academic and industrial researchers to present new work on neural methods for retrieval. In addition, a special track was organized focusing on resources for evaluation and reproducibility, including proposals for public benchmarking datasets and shared model repositories. A total of 19 papers?which included five special track papers? were presented in the form of oral or poster presentations. Organizers of four of the TREC 2017 tracks were invited to present at the workshop on how these IR tasks may be suitable for evaluating recent data-hungry neural approaches. The full-day workshop?with more than 170 registrants?concluded with an engaging panel discussion.

Adaptive domain modelling for information retrieval

Modern search engines employ a number of interactive features to assist users in exploring the document collection and expressing their information needs. Providing these features require knowledge about the document collection in the domain, i.e. a domain model. These models may be difficult to obtain and even if they are available, they may become out of date when the document collection changes or the users start to view the domain differently. In this thesis we propose to use implicit feedback left by users while they interact with search engines to build domain models that evolve over time. These models can adapt to changes in the domain as reflected in the search trend of the user population. We validate these models in two different IR tasks. The major application is query recommendation where previous studies focused on Web search in general or did not consider the temporal aspect of recommendations. Our models address these issues as they are targeted to specific domains such as enterprise search or digital libraries. We furthermore devise an automatic evaluation methodology that allows us to perform extensive evaluation of our adaptive models and observe their performance over time. Using query logs collected from two academic institutions, the evaluation framework assesses the impact of different factors on their performance. The second application of these models is query session retrieval. The query session retrieval problem extends the traditional ad-hoc retrieval by taking into account the previous user interactions with the retrieval system within the same session when answering the query. In this context rather than explicitly providing the user with relevant queries to their information needs, our adaptive models implicitly derive query expansions relevant to the user information needs as identified in the sessions by mapping the session to similar sessions inferred from the models.

Exploring topic structure

The use of topical information has long been studied in the context of information retrieval. For example, grouping search results into topical categories enables more effective information presentation to users, while grouping documents in a collection can lead to efficient information access. We define a topic as the main theme or subject contained in a (set of) document(s). While topics provide information about the subjects contained in a document, the structure of topics provides information such as the degree to which a set of documents is focused on certain topic (or set of topics), topical diversity among documents, and semantic relatedness of topics. The work of this thesis focuses on modeling the structure of topics present in a (set of) document(s), with the goal of effectively using it in information retrieval. In particular, we consider a number of IR tasks where the notion of relevance is beyond "aboutness" and topic structure plays an important role in satisfying users' information need. The following research themes are addressed, in three parts: (1) topic coherence, (2) diversity and the cluster hypothesis, and (3) relating topics present in different representations. With respect to the first research theme, we develop a coherence measure that effectively captures topical coherence of a set of documents. The proposed measure is applied to two IR tasks, namely, blog feed retrieval and query performance prediction. In the second part of the thesis, we explore the impact of topic structure on effectively presenting retrieval results, with a focus on the scenario of result diversification. We re-visit the cluster hypothesis with respect to ambiguous or multi-faceted queries and investigate the effectiveness of query-specific clustering in result diversification. Topics can be represented in different ways, e.g., using clusters, using definitions from a thesaurus, using statistics of term frequencies, etc. In the third part of the thesis, we study the problem of relating topics represented in different forms within the context of automatic link generation. We identify a set of significant terms from a source text, link those terms to their corresponding entries in a knowledge base in such a way that the source text is annotated with background information available in the knowledge base. We conduct a case study in automatically generating links from narrative radiology reports to Wikipedia. Such links are expected to help users understand the medical terminology and thereby increase the value of the reports. Here we evaluate state-of-the-art link generation systems and propose an approach that improves over state-of-the-art systems on radiology data. Available online at http://dare.uva.nl/record/377895.

Improving document clustering using Okapi BM25 feature weighting

We investigate the effect of feature weighting on document clustering, including a novel investigation of Okapi BM25 feature weighting. Using eight document datasets and 17 well-established clustering algorithms we show that the benefit of tf-idf weighting over tf weighting is heavily dependent on both the dataset being clustered and the algorithm used. In addition, binary weighting is shown to be consistently inferior to both tf-idf weighting and tf weighting. We investigate clustering using both BM25 term saturation in isolation and BM25 term saturation with idf, confirming that both are superior to their non-BM25 counterparts under several common clustering quality measures. Finally, we investigate estimation of the k1 BM25 parameter when clustering. Our results indicate that typical values of k1 from other IR tasks are not appropriate for clustering; k1 needs to be higher.

Social bookmark weighting for search and recommendation

Social bookmarking enables knowledge sharing and efficient discovery on the web, where users can collaborate together by tagging documents of interests. A lot of attention was given lately for utilizing social bookmarking data to enhance traditional IR tasks. Yet, much less attention was given to the problem of estimating the effectiveness of an individual bookmark for the specific tasks. In this work, we propose a novel framework for social bookmark weighting which allows us to estimate the effectiveness of each of the bookmarks individually for several IR tasks. We show that by weighting bookmarks according to their estimated quality, we can significantly improve social search effectiveness. We further demonstrate that using the same framework, we can derive solutions to several recommendation tasks such as tag recommendation, user recommendation, and document recommendation. Empirical evaluation on real data gathered from two large bookmarking systems demonstrates the effectiveness of the new social bookmark weighting framework.

People search in the enterprise

The enormous increase in recent years in the amount of information available online has led to a renewed interest in a broad range of IR-related areas that go beyond plain document retrieval. Some of this new attention has fallen on a subset of IR tasks, in particular on entity retrieval tasks. This emerging area differs from traditional document retrieval in a number of ways. Entities are not represented directly (as retrievable units such as documents), and we need to identify them "indirectly" through occurrences in documents. This brings new, exciting challenges to the fields of Information Retrieval and Information Extraction. In this thesis we focus on one particular type of entity: people.

The diversity-based approach to open-domain text summarization

The paper introduces a novel approach to unsupervised text summarization, which in principle should work for any domain or genre. The novelty lies in exploiting the diversity of concepts in text for summarization, which has not received much attention in the summarization literature. We propose, in addition, what we call the information-centric approach to evaluation, where the quality of summaries is judged not in terms of how well they match human-created summaries but in terms of how well they represent their source documents in IR tasks such document retrieval and text categorization. To find the effectiveness of our approach under the proposed evaluation scheme, we set out to examine how a system with the diversity functionality performs against one without, using the test data known as BMIR-J2. The results demonstrate a clear superiority of the diversity-based approach to a non-diversity-based approach. The paper also addresses the question of how closely the diversity approach models human judgments on summarization. We have created a relatively large volume of data annotated for relevance to summarization by human subjects. We have trained a decision tree-based summarizer using the data, and examined how the diversity method compares with the supervised method in performance when tested on the data. It was found that the diversity approach performs as well as and in some cases superior to the supervised method.