Information Retrieval Community Research Articles

Eliminating Negative Word Similarities for Measuring Document Distances: A Thoroughly Empirical Study on Word Mover's Distance.

Document distance is a fundamental yet significant research topic in the information retrieval community, and its accuracy dominates the performance of many text retrieval applications. Beyond the Bag-of-Words (BoW) model, the Word Mover's Distance (WMD) semantically defines the distance between documents as the minimum cost (i.e., measured by word similarities of embeddings) required to transport the words from one document to another, and it has been proven to be superior by k -nearest neighbor classification. In this article, we thoroughly study the characteristics of WMD and its relaxed versions, e.g., Relaxed WMD (RWMD) and Iterative Constrained Transfers (ICT), in various scenarios. Specifically, we concentrate on the problem of negative word similarity: the WMD family leverages all word similarities, however, most of them are meaningless, resulting in negative effects for measuring document distances. To remedy this problem, we propose Informative Similarity Filter (ISF), which retains a very small percentage of top word similarities and fixes the others as the same lower similarity. Built on it, we propose a greedy optimization (GOM) for WMD, an accurate approximation to WMD. We theoretically analyze that ISF-GOM is more applicable for relatively longer documents. Extensive experiments have been conducted to validate: 1) the problem of RWMD; 2) the effectiveness of ISF-GOM; and 3) the consistence of our analysis of ISF-GOM. Our codes and datasets are available at https://github.com/BoCheng-96/ISF-GOM.

Graph-based Text Classification by Contrastive Learning with Text-level Graph Augmentation

Text Classification (TC) is a fundamental task in the information retrieval community. Nowadays, the mainstay TC methods are built on the deep neural networks, which can learn much more discriminative text features than the traditional shallow learning methods. Among existing deep TC methods, the ones based on Graph Neural Network (GNN) have attracted more attention due to the superior performance. Technically, the GNN-based TC methods mainly transform the full training dataset to a graph of texts; however, they often neglect the dependency between words, so as to miss potential semantic information of texts, which may be significant to exactly represent them. To solve the aforementioned problem, we generate graphs of words instead, so as to capture the dependency information of words. Specifically, each text is translated into a graph of words, where neighboring words are linked. We learn the node features of words by a GNN-like procedure and then aggregate them as the graph feature to represent the current text. To further improve the text representations, we suggest a contrastive learning regularization term. Specifically, we generate two augmented text graphs for each original text graph, we constrain the representations of the two augmented graphs from the same text close and the ones from different texts far away. We propose various techniques to generate the augmented graphs. Upon those ideas, we develop a novel deep TC model, namely Text-level Graph Networks with Contrastive Learning (TGN cl ). We conduct a number of experiments to evaluate the proposed TGN cl model. The empirical results demonstrate that TGN cl can outperform the existing state-of-the-art TC models.

Report on the 1st Workshop on Task Focused IR in the Era of Generative AI

Search and recommender systems should prioritize support for user tasks over support for individual queries or actions. To that end, the Information Retrieval (IR) community has spent decades trying to understand users' tasks and their contexts, and how to best assist users in making progress toward completing them. The recent advancements in generative artificial intelligence (AI) have drastically shifted the landscape of task-focused IR. Users can now express not only their queries and questions, but actual information needs, tasks, and goals in natural language to an AI system and receive not just results, but also answers in natural language that are generated specifically for them. This new paradigm raises many interesting questions, opportunities, and challenges. We brought together a group of highly motivated students and scholars in a two-day workshop on the Microsoft campus in Redmond to discuss these issues, learn from each other, and envision a new future for task-focused IR and information access more broadly. Date : 28--29 September 2023. Website : https://ir-ai.github.io.

Computational Technologies for Fashion Recommendation: A Survey

Fashion recommendation is a key research field in computational fashion research and has attracted considerable interest in the computer vision, multimedia, and information retrieval communities in recent years. Due to the great demand for applications, various fashion recommendation tasks, such as personalized fashion product recommendation, complementary (mix-and-match) recommendation, and outfit recommendation, have been posed and explored in the literature. The continuing research attention and advances impel us to look back and in-depth into the field for a better understanding. In this article, we comprehensively review recent research efforts on fashion recommendation from a technological perspective. We first introduce fashion recommendation at a macro level and analyze its characteristics and differences with general recommendation tasks. We then clearly categorize different fashion recommendation efforts into several sub-tasks and focus on each sub-task in terms of its problem formulation, research focus, state-of-the-art methods, and limitations. We also summarize the datasets proposed in the literature for use in fashion recommendation studies to give readers a brief illustration. Finally, we discuss several promising directions for future research in this field. Overall, this survey systematically reviews the development of fashion recommendation research. It also discusses the current limitations and gaps between academic research and the real needs of the fashion industry. In the process, we offer a deep insight into how the fashion industry could benefit from the computational technologies of fashion recommendation.

A semantically enhanced text retrieval framework with abstractive summarization

AbstractRecently, large pretrained language models (PLMs) have led a revolution in the information retrieval community. In most PLMs‐based retrieval frameworks, the ranking performance broadly depends on the model structure and the semantic complexity of the input text. Sequence‐to‐sequence generative models for question answering or text generation have proven to be competitive, so we wonder whether these models can improve ranking effectiveness by enhancing input semantics. This article introduces SE‐BERT, a semantically enhanced bidirectional encoder representation from transformers (BERT) based ranking framework that captures more semantic information by modifying the input text. SE‐BERT utilizes a pretrained generative language model to summarize both sides of the candidate passage and concatenate them into a new input sequence, allowing BERT to acquire more semantic information within the constraints of the input sequence's length. Experimental results from two Text Retrieval Conference datasets demonstrate that our approach's effectiveness increasing as the length of the input text increases.

Into the Unknown: Exploration of Search Engines’ Responses to Users with Depression and Anxiety

Researchers worldwide have explored the behavioral nuances that emerge from interactions of individuals afflicted by mental health disorders (MHD) with persuasive technologies, mainly social media. Yet, there is a gap in the analysis pertaining to a persuasive technology that is part of their everyday lives: web search engines (SE). Each day, users with MHD embark on information seeking journeys using popular SE, like Google or Bing. Every step of the search process for better or worse has the potential to influence a searcher’s mindset. In this work, we empirically investigate what subliminal stimulus SE present to these vulnerable individuals during their searches. For this, we use synthetic queries to produce associated query suggestions and search engine results pages. Then we infer the subliminal stimulus present in text from SE, i.e., query suggestions, snippets, and web resources. Findings from our empirical analysis reveal that the subliminal stimulus displayed by SE at different stages of the information seeking process differ between MHD searchers and our control group composed of “average” SE users. Outcomes from this work showcase open problems related to query suggestions, search engine result pages, and ranking that the information retrieval community needs to address so that SE can better support individuals with MHD.

Semi-Supervised Lexicon-Aware Embedding for News Article Time Estimation

In the information retrieval community, Temporal Information Retrieval (TIR) has become increasingly popular. Documents focused on the time surrounding their publication are more likely to be accurate and contain information relevant to the reader. In this study, we explore the inverted pyramid paradigm by extracting temporal expressions from news documents, standardizing their values, and evaluating them based on their position within the text. We present a lexicon expansion method that employs WordNet as input. This approach enhances the lexicon by grouping words with similar meanings, potentially improving the accuracy of event detection algorithms. Additionally, this process can introduce new words and phrases to the lexicon, expanding the vocabulary. Using each tagged dataset, a classifier is trained with a pre-trained network. A pool of unlabeled data are processed, and high-confidence pseudo-labels are assigned. Pseudo-labels are generated by leveraging the partially trained model and the original labelled data. As the classifier predicts the correct label for a data sample, the pseudo-labels of other data samples are updated, and vice versa. At the end of this process, the predictions from different matching classifiers are combined. It takes several rounds to label the unlabeled inputs using this method. To evaluate the proposed solutions, we conducted experiments on 4,500 online news articles relevant to temporal retrieval. LSTM, BiLSTM, and BERT models with and without lexicon expansion were assessed based on log loss and relative divergence of entropy. A jointly trained semi-supervised learning model achieved a mean KL divergence of 0.89, an F1 score of 0.74 for temporal events, and 0.63 for non-temporal events. Besides alleviating data sparsity issues and enabling the training of more complex networks, this technique can also serve as an alternative to data augmentation methods.

Information Retrieval meets Large Language Models: A strategic report from Chinese IR community

The research field of Information Retrieval (IR) has evolved significantly, expanding beyond traditional search to meet diverse user information needs. Recently, Large Language Models (LLMs) have demonstrated exceptional capabilities in text understanding, generation, and knowledge inference, opening up exciting avenues for IR research. LLMs not only facilitate generative retrieval but also offer improved solutions for user understanding, model evaluation, and user-system interactions. More importantly, the synergistic relationship among IR models, LLMs, and humans forms a new technical paradigm that is more powerful for information seeking. IR models provide real-time and relevant information, LLMs contribute internal knowledge, and humans play a central role of demanders and evaluators to the reliability of information services. Nevertheless, significant challenges exist, including computational costs, credibility concerns, domain-specific limitations, and ethical considerations. To thoroughly discuss the transformative impact of LLMs on IR research, the Chinese IR community conducted a strategic workshop in April 2023, yielding valuable insights. This paper provides a summary of the workshop’s outcomes, including the rethinking of IR’s core values, the mutual enhancement of LLMs and IR, the proposal of a novel IR technical paradigm, and open challenges.

Are Neural Ranking Models Robust?

Recently, we have witnessed the bloom of neural ranking models in the information retrieval (IR) field. So far, much effort has been devoted to developing effective neural ranking models that can generalize well on new data. There has been less attention paid to the robustness perspective. Unlike the effectiveness, which is about the average performance of a system under normal purpose, robustness cares more about the system performance in the worst case or under malicious operations instead. When a new technique enters into the real-world application, it is critical to know not only how it works in average, but also how would it behave in abnormal situations. So, we raise the question in this work: Are neural ranking models robust? To answer this question, first, we need to clarify what we refer to when we talk about the robustness of ranking models in IR. We show that robustness is actually a multi-dimensional concept and there are three ways to define it in IR: (1) the performance variance under the independent and identically distributed (I.I.D.) setting; (2) the out-of-distribution (OOD) generalizability ; and (3) the defensive ability against adversarial operations. The latter two definitions can be further specified into two different perspectives, respectively, leading to five robustness tasks in total. Based on this taxonomy, we build corresponding benchmark datasets, design empirical experiments, and systematically analyze the robustness of several representative neural ranking models against traditional probabilistic ranking models and learning-to-rank (LTR) models. The empirical results show that there is no simple answer to our question. While neural ranking models are less robust against other IR models in most cases, some of them can still win two out of five tasks. This is the first comprehensive study on the robustness of neural ranking models. We believe the way we study the robustness as well as our findings would be beneficial to the IR community. We will also release all the data and codes to facilitate the future research in this direction.

Report on the 2nd Joint Conference of the Information Retrieval Communities in Europe (CIRCLE 2022)

The Joint Conference of the Information Retrieval Communities in Europe (CIRCLE 2020) is the second joint conference of the French, Italian, and Spanish information retrieval communities. The three days of conference gathered interesting studies and research work on a wide range of topics on information retrieval, such as topic and document modeling, web search, information retrieval in e-government, social media, recommender systems, information retrieval evaluation, indexing and annotation, user profiling and interaction, frameworks and systems, and semantic extraction. It was hosted by Université de Toulouse, France in a holiday resort at Samatan. Date: 4--7 July, 2022. Website: https://www.irit.fr/CIRCLE/.

Transferring knowledge between topics in systematic reviews

In the medical domain, a systematic review (SR) is a well-structured process aimed to review all available literature on a research question. This is however a laborious task, both in terms of money and time. As such, the automation of a SR with the aid of technology has received interest in several research communities, among which the Information Retrieval community. In this work, we experiment on the possibility of leveraging previously conducted systematic reviews to train a classifier/ranker which is later applied to a new SR. We also investigate on the possibility of pre-training Deep Learning models and eventually tuning them in an Active Learning process. Our results show that the pre-training of these models deliver a good zero-shot (i.e., with no fine-tuning) ranking, achieving an improvement of 79% for the MAP metric, with respect to a standard classifier trained on few in-domain documents. However, the pre-trained deep learning algorithms fail to deliver consistent results when continuously trained in an Active Learning scenario: our analysis shows that using smaller sized models and employing adapter modules might enable an effective active learning training.

Analyzing the Information Content of Text-Based Files in Supplementary Materials of Biomedical Literature.

We present an analysis of supplementary materials of PubMed Central (PMC) articles and show their importance in indexing and searching biomedical literature, in particular for the emerging genomic medicine field. On a subset of articles from PubMed Central, we use text mining methods to extract MeSH terms from abstracts, full texts, and text-based supplementary materials. We find that the recall of MeSH annotations increases by about 5.9 percentage points (+20% on relative percentage) when considering supplementary materials compared to using only abstracts. We further compare the supplementary material annotations with full-text annotations and we find out that the recall of MeSH terms increases by 1.5 percentage point (+3% on relative percentage). Additionally, we analyze genetic variant mentions in abstracts and full-texts and compare them with mentions found in supplementary text-based files. We find that the majority (about 99%) of variants are found in text-based supplementary files. In conclusion, we suggest that supplementary data should receive more attention from the information retrieval community, in particular in life and health sciences.

Perceptions of Diversity in Electronic Music: the Impact of Listener, Artist, and Track Characteristics

Shared practices to assess the diversity of retrieval system results are still debated in the Information Retrieval community, partly because of the challenges of determining what diversity means in specific scenarios, and of understanding how diversity is perceived by end-users. The field of Music Information Retrieval is not exempt from this issue. Even if fields such as Musicology or Sociology of Music have a long tradition in questioning the representation and the impact of diversity in cultural environments, such knowledge has not been yet embedded into the design and development of music technologies. In this paper, focusing on electronic music, we investigate the characteristics of listeners, artists, and tracks that are influential in the perception of diversity. Specifically, we center our attention on 1) understanding the relationship between perceived diversity and computational methods to measure diversity, and 2) analyzing how listeners' domain knowledge and familiarity influence such perceived diversity. To accomplish this, we design a user-study in which listeners are asked to compare pairs of lists of tracks and artists, and to select the most diverse list from each pair. We compare participants' ratings with results obtained through computational models built using audio tracks' features and artist attributes. We find that such models are generally aligned with participants' choices when most of them agree that one list is more diverse than the other, while they present a mixed behaviour in cases where participants have little agreement. Moreover, we observe how differences in domain knowledge, familiarity, and demographics can influence the level of agreement among listeners, and between listeners and diversity metrics computed automatically.

Simulating and Modeling the Risk of Conversational Search

In conversational search, agents can interact with users by asking clarifying questions to increase their chance of finding better results. Many recent works and shared tasks in both natural language processing and information retrieval communities have focused on identifying the need to ask clarifying questions and methodologies of generating them. These works assume that asking a clarifying question is a safe alternative to retrieving results. As existing conversational search models are far from perfect, it is possible and common that they could retrieve/generate bad clarifying questions. Asking too many clarifying questions can also drain a user’s patience when the user prefers searching efficiency over correctness. Hence, these models can backfire and harm a user’s search experience due to these risks from asking clarifying questions. In this work, we propose a simulation framework to simulate the risk of asking questions in conversational search and further revise a risk-aware conversational search model to control the risk. We show the model’s robustness and effectiveness through extensive experiments on three conversational datasets — MSDialog, Ubuntu Dialog Corpus, and Opendialkg — in which we compare it with multiple baselines. We show that the risk-control module can work with two different re-ranker models and outperform all of the baselines in most of our experiments.

Deep Listwise Triplet Hashing for Fine-Grained Image Retrieval.

Hashing is a practical approach for the approximate nearest neighbor search. Deep hashing methods, which train deep networks to generate compact and similarity-preserving binary codes for entities (e.g. images), have received lots of attention in the information retrieval community. A representative stream of deep hashing methods is triplet-based hashing that learns hashing models from triplets of data. The existing triplet-based hashing methods only consider triplets that are in the form of (q,q+,q-) , where q and q+ are in the same class and q and q- are in different classes. However, the number of possible triplets is approximately the cube of training examples, triplets used in the existing methods are only a small fraction of all possible triplets. This motivates us to develop a new triplet-based hashing method that adopts many more triplets in training phase. We propose Deep Listwise Triplet Hashing (DLTH) that introduces more triplets into batch-based training and a novel listwise triplet loss to capture the relative similarity in new triplets. This method has a pipeline of two steps. In Step 1, we propose a novel way to generate triplets from the soft class labels obtained by knowledge distillation module, where the triplets in the form of (q,q+,q-) are a subset of the newly obtained triplets. In Step 2, we develop a novel listwise triplet loss to train the hashing network, which seeks to capture the relative similarity between images in triplets according to soft labels. We conduct comprehensive image retrieval experiments on four benchmark datasets. The experimental results show that the proposed method has superior performances over state-of-the-art baselines.

Batch Evaluation Metrics in Information Retrieval: Measures, Scales, and Meaning

A sequence of recent papers, including in this journal, has considered the role of measurement scales in information retrieval (IR) experimentation, and presented the argument that (only) uniform-step interval scales should be used. Hence, it has been argued, well-known metrics such as reciprocal rank, expected reciprocal rank, normalized discounted cumulative gain, and average precision, should be either discarded as measurement tools, or adapted so that their metric values lie at uniformly-spaced points on the number line. These papers paint a rather bleak picture of past decades of IR evaluation, at odds with the IR community’s overall emphasis on practical experimentation and measurable improvement. Our purpose in this work is to challenge that pessimistic assessment. In particular, we argue that mappings from categorical and ordinal data to sets of points on the number line are valid provided there is an external reason for each target point to have been selected. We first consider the general role of measurement scales, and of categorical, ordinal, interval, ratio, and absolute data collections. In connection with the first two of those categories we also provide examples of the knowledge that is captured and represented by numeric mappings to the real number line. Focusing then on information retrieval, we argue that document rankings are categorical data, and that the role of an effectiveness metric is to provide a single value that summarizes the usefulness to a user or population of users of any given ranking, with usefulness able to be represented as a continuous variable on a ratio scale. That is, we argue that most current IR metrics are well-founded, and, moreover, that those metrics are more meaningful in their current form than in the proposed “intervalized” versions.

A Mathematical Method with Word Frequency Sorting for Mining Web Content Outliers

The Internet is a massive collection of information that makes it extremely difficult to search and retrieve the required and valuable information. So, Search engine became an important tool for searching various data from the web. The primary evaluation of search engine is effectiveness and efficiency. While searching for information through search engines, always users retrieve redundant and irrelevant information. This replicated and uninteresting information affects both the effectiveness and efficiency of search engine by wasting users’ time by browsing the uninterested documents and its accessing time. Web content outlier mining plays a significant role in identifying and removing these redundant document (outliers) which is an important issue among the information retrieval and web mining research communities since most of the people rely on search engines to get the required information. Most existing algorithms for web content outlier mining focuses attention on applying weightage only to the common terms in the documents whereas in this research work, a mathematical approach based on term frequency ranking to identify the duplicates, and uses the domain dictionary to check for relevant document has been carried out to improve the effectiveness and efficiency of the search engine.

Link Prediction Between Structured Geopolitical Events: Models and Experiments.

Often thought of as higher-order entities, events have recently become important subjects of research in the computational sciences, including within complex systems and natural language processing (NLP). One such application is event link prediction. Given an input event, event link prediction is the problem of retrieving a relevant set of events, similar to the problem of retrieving relevant documents on the Web in response to keyword queries. Since geopolitical events have complex semantics, it is an open question as to how to best model and represent events within the framework of event link prediction. In this paper, we formalize the problem and discuss how established representation learning algorithms from the machine learning community could potentially be applied to it. We then conduct a detailed empirical study on the Global Terrorism Database (GTD) using a set of metrics inspired by the information retrieval community. Our results show that, while there is considerable signal in both network-theoretic and text-centric models of the problem, classic text-only models such as bag-of-words prove surprisingly difficult to outperform. Our results establish both a baseline for event link prediction on GTD, and currently outstanding challenges for the research community to tackle in this space.

Diversity by Design in Music Recommender Systems

Music Recommender Systems (Music RS) are nowadays pivotal in shaping the listening experience of people all around the world. Partly driven by the commercial application of this technology, music recommendation research has gained increasing attention both within and outside the Music Information Retrieval (MIR) community. Thanks also to the widespread use of recommender systems in music streaming services, it has been possible to enhance several characteristics of such systems in terms of performance, design, and user experience. Nonetheless, imagining Music RS only from an application-driven perspective may generate an incomplete view of how this technology is affecting people’s habitus, from the decision-making processes to the formation of musical taste and opinions. In this overview, we address the concept of diversity in music recommendation, and taking a value-driven approach we review diversity-related methodologies proposed in the Music RS literature. Additionally, by taking as an example the wider context of Information Technology (IT), we present the elements interacting in the diversity by design paradigm. We do that to acknowledge the lack of a comprehensive framework in Music RS research to address diversity, until now mostly driven by empirical results and fragmented in different application areas. Maintaining an interdisciplinary perspective, we discuss some challenges that MIR practitioners may face when researching Music RS, going beyond the search for better performance and instead questioning the theoretical foundations on which to base future research.

A Bregman Divergence Optimization Framework for Ranking on Data Manifold and Its New Extensions

Recently, graph-based ranking algorithms have received considerable interests in machine learning, computer vision and information retrieval communities. Ranking on data manifold (or manifold ranking, MR) is one of the representative approaches. One of the limitations of manifold ranking is its high computational complexity (O(n3), where n is the number of samples in database). In this paper, we cast the manifold ranking into a Bregman divergence optimization framework under which we transform the original MR to an equivalent optimal kernel matrix learning problem.With this new formulation, two effective and efficient extensions are proposed to enhance the ranking performance. Extensive experimental results on two real world image databases show the effectiveness of the proposed approach.