Information Retrieval Research Research Articles

Information Retrieval meets Large Language Models: A strategic report from Chinese IR community

The research field of Information Retrieval (IR) has evolved significantly, expanding beyond traditional search to meet diverse user information needs. Recently, Large Language Models (LLMs) have demonstrated exceptional capabilities in text understanding, generation, and knowledge inference, opening up exciting avenues for IR research. LLMs not only facilitate generative retrieval but also offer improved solutions for user understanding, model evaluation, and user-system interactions. More importantly, the synergistic relationship among IR models, LLMs, and humans forms a new technical paradigm that is more powerful for information seeking. IR models provide real-time and relevant information, LLMs contribute internal knowledge, and humans play a central role of demanders and evaluators to the reliability of information services. Nevertheless, significant challenges exist, including computational costs, credibility concerns, domain-specific limitations, and ethical considerations. To thoroughly discuss the transformative impact of LLMs on IR research, the Chinese IR community conducted a strategic workshop in April 2023, yielding valuable insights. This paper provides a summary of the workshop’s outcomes, including the rethinking of IR’s core values, the mutual enhancement of LLMs and IR, the proposal of a novel IR technical paradigm, and open challenges.

Understanding Feature Importance in Musical Works: Unpacking Predictive Contributions to Cluster Analyses

Cluster analysis provides insight into musical patterns in composition, performance, and perception. Despite its wide adoption in music research, understanding how specific features affect clustering solutions remains challenging. For example, features such as mode (i.e., major/minor), timing, signal amplitude, and pitch are often intercorrelated, making it difficult to understand their specific role within different clusters. To demonstrate how accumulated local effects (ALEs) can help with this challenge, here we analyze 48 excerpts from complete sets of preludes by Bach and Chopin, showing how specific features contribute to two- and three-cluster analyses. These exploratory analyses reveal that ALEs can identify salient or subtle data patterns from cluster analyses by tracking how changes in features affect cluster membership. We explore these insights in visualizations quantifying feature importance and an interactive companion application ( https://maplelab.net/feature-importance/ ) featuring the analyzed audio. Following a demonstration of this method, we suggest how it can be applied to explore topics of interest to researchers in music information retrieval, empirical musicology, and music cognition alike.

A Probabilistic Model Toward How People Search to Build Outcomes

Although increased attention is being given to understanding how people search to build task outcomes, a formal model of the relation between how people search and how people build task outcomes is still lacking. This paper proposes a unified probabilistic model of how people search to build outcomes. The model involves 3 types of searcher behaviors (i.e., query submission, document selection, and information transformation) to model the effect of the information collected during search, and uses the item response theory to capture the ternary relations between the ability to transform information, the information collected, and the probability of successfully building task outcomes. We evaluate the proposed model in the task of identifying searchers’ proficiencies under the assumption that high proficiency searchers would have high abilities to transform information. The results obtained high accuracies and F1 scores, which could reflect the effectiveness of the proposed model. The model contributes to the formal understanding of how people search to build task outcomes, and provides new possibilities for personalized and session-based information retrieval research.

Characterizing Elite Scholars of Library and Information Science: A Bibliometric Analysis

Publication metrics have become crucial to measure the productivity and impact of individuals and organizations for highlighting their influence upon the knowledge world. This study aims to present a bibliometric analysis of the top two percent highly cited Library and Information Science (LIS) researchers in the Stanford University list of the top two percent researchers of various disciplines published in 2021. We have used the bibliometric method to analyze 254 highly cited LIS researchers. The findings show male dominance and female underrepresentation in LIS research. Leydesdorff, Loet is the most highly cited researcher of the LIS field. The Amsterdam School of Communications Research, Netherlands, is the most highly cited organization of LIS based on the highly cited researchers affiliated with it. The majority of LIS researchers belong to USA and UK. There is a meager contribution of Asian, Australian, and African countries producing highly cited LIS researchers. Highly cited LIS researchers are more familiar with publishing research in information retrieval, information seeking, and bibliometrics. Positions of researchers at different ranks by different measures depict their productivity, impact, h- index, and a composite score. A strong correlation exists between total publications and total citations. The study, in its nature, is unique and will provide an insight for young and emerging researchers for selection of research direction, persons for research collaboration, and a motivation to invest more efforts for impactful research.

Reflection on future directions: a systematic review of reported limitations and solutions in interactive information retrieval user studies

PurposeThe purpose of this paper is to understand how users behave and evaluate how systems with users are essential for interactive information retrieval (IIR) research. User study methodology serves as a primary approach to answering IIR research questions. In addition to designing user study procedures, understanding the limitations of varying study designs and discussing solutions to the limitations is also critical for improving the methods and advancing the knowledge in IIR.Design/methodology/approachGiven this unresolved gap, we apply the faceted framework developed by Liu and Shah (2019) in systematically reviewing 131 IIR user studies recently published (2016–2021) on multiple IR and information science venues.FindingsOur study achieve three goals: (1) extracting and synthesizing the reported limitations on multiple aspects of user study (e.g. recruitment, tasks, study procedures, system interfaces, data analysis methods) under associated facets; (2) summarizing the reported solutions to the limitations; (3) clarifying the connections between types of limitations and types of solutions.Practical implicationsThe bibliography of user studies can be used by students and junior researchers who are new to user-centered IR studies as references for study design. Our results can facilitate the reflection and improvement on IR research methodology and serve as a checklist for evaluating customized IIR user studies in varying problem spaces.Originality/valueTo our knowledge, this work is the first study that systematically reviews the study limitations and solutions reported by IIR researchers in articles and empirically examines their connections to different study components.

A Framework for Enabling Unpaired Multi-Modal Learning for Deep Cross-Modal Hashing Retrieval.

Cross-Modal Hashing (CMH) retrieval methods have garnered increasing attention within the information retrieval research community due to their capability to deal with large amounts of data thanks to the computational efficiency of hash-based methods. To date, the focus of cross-modal hashing methods has been on training with paired data. Paired data refers to samples with one-to-one correspondence across modalities, e.g., image and text pairs where the text sample describes the image. However, real-world applications produce unpaired data that cannot be utilised by most current CMH methods during the training process. Models that can learn from unpaired data are crucial for real-world applications such as cross-modal neural information retrieval where paired data is limited or not available to train the model. This paper provides (1) an overview of the CMH methods when applied to unpaired datasets, (2) proposes a framework that enables pairwise-constrained CMH methods to train with unpaired samples, and (3) evaluates the performance of state-of-the-art CMH methods across different pairing scenarios.

ACM SIG IR Annual Business Meeting 2022

The 2022 ACM SIG IR Annual Business Meeting took place as a hybrid event during the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval in Madrid, Spain on July 14, 2022. The business meeting was led by the chair of the SIG IR Executive Committee (EC), Vanessa Murdock.

Report on the 1st Workshop on Reaching Efficiency in Neural Information Retrieval (ReNeuIR 2022) at SIGIR 2022

As Information Retrieval (IR) researchers, we not only develop algorithmic solutions to hard problems, but we also insist on a proper, multifaceted evaluation of ideas. The IR literature on the fundamental topic of retrieval and ranking, for instance, has a rich history of studying the effectiveness of indexes, retrieval algorithms, and complex machine learning rankers and, at the same time, quantifying their computational costs, from creation and training to application and inference. This is evidenced, for example, by more than a decade of research on efficient training and inference of large decision forest models in Learning to Rank (LtR). As we move towards even more complex, deep learning models in a wide range of applications, questions on efficiency have once again become relevant with renewed urgency. Indeed, efficiency is no longer limited to time- and space-efficiency; instead it has found new, challenging dimensions that stretch to resource-, sample- and energy-efficiency with ramifications for researchers, users, and the environment. As a step towards bringing together experts from industry and academia and creating a forum for a critical discussion and the promotion of efficiency in the era of Neural Information Retrieval (NIR), we held the ReNeuIR workshop on July 15 th , 2022 as a hybrid event---in person in Madrid, Spain along with online attendees---in conjunction with ACM SIGIR 2022. Recognizing the importance of this topic, approximately 80 participants answered our call and attended the workshop over three sessions. The event included a total of two keynotes and eight paper presentations, and concluded with a lively discussion where participants helped identify gaps in existing research and brainstormed future research directions. We had consensus in recognizing that efficiency is not simply latency, that a holistic, concrete definition of efficiency is needed to guide researchers and reviewers alike, and that more research is necessary in the development of efficiency-centered evaluation metrics and standard benchmark datasets, platforms, and tools. Date: 15 July, 2022. Website: https://ReNeuIR.org.

Musical Success in the United States and Brazil: Novel Datasets and Temporal Analyses

Music is not only a worldwide essential cultural industry but also one of the most dynamic. The increasing volume of complex music-related data defines new challenges and opportunities for extracting knowledge, benefiting not only different music segments but also the Music Information Retrieval research field. In this article, we assess musical success in the United States and Brazil, two of the biggest music markets in the world. We first introduce MUHSIC and MUHSIC-BR, two novel datasets with enhanced success information that combine chart-related data with acoustic metadata to describe the temporal evolution of musical careers. Then, we use such enriched and curated data to cluster artists according to their success level by considering their high-impact periods (hot streaks). Our results reveal three groups with distinct success behavior over time. Furthermore, Brazil and the US present specific music success patterns regarding artists and genres, reflecting the importance of analyzing regional markets individually.

Relevance assessments, bibliometrics, and altmetrics: a quantitative study on PubMed and arXiv

Relevance is a key element for analyzing bibliometrics and information retrieval (IR). In both domains, relevance decisions are discussed theoretically and sometimes evaluated in empirical studies. IR research is often based on test collections for which explicit relevance judgments are made, while bibliometrics is based on implicit relevance signals like citations or other non-traditional quantifiers like altmetrics. While both types of relevance decisions share common concepts, it has not been empirically investigated how they relate to each other on a larger scale. In this work, we compile a new dataset that aligns IR relevance judgments with traditional bibliometric relevance signals (and altmetrics) for life sciences and physics publications. The dataset covers PubMed and arXiv articles, for which relevance judgments are taken from TREC Precision Medicine and iSearch, respectively. It is augmented with bibliometric data from the Web of Science and Altmetrics. Based on the reviewed literature, we outline a mental framework supporting the answers to our research questions. Our empirical analysis shows that bibliometric (implicit) and IR (explicit) relevance signals are correlated. Likewise, there is a high correlation between biblio- and altmetrics, especially for documents with explicit positive relevance judgments. Furthermore, our cross-domain analysis demonstrates the presence of these relations in both research fields.

Meta-search based approach for Arabic information retrieval

PurposeThe semantic relations between Arabic word representations were recognized and widely studied in theoretical studies in linguistics many centuries ago. Nonetheless, most of the previous research in automatic information retrieval (IR) focused on stem or root-based indexing, while lemmas and patterns are under-exploited. However, the authors believe that each of the four morphological levels encapsulates a part of the meaning of words. That is, the purpose is to aggregate these levels using more sophisticated approaches to reach the optimal combination which enhances IR.Design/methodology/approachThe authors first compare the state-of-the art Arabic natural language processing (NLP) tools in IR. This allows to select the most accurate tool in each representation level i.e. developing four basic IR systems. Then, the authors compare two rank aggregation approaches which combine the results of these systems. The first approach is based on linear combination, while the second exploits classification-based meta-search.FindingsCombining different word representation levels, consistently and significantly enhances IR results. The proposed classification-based approach outperforms linear combination and all the basic systems.Research limitations/implicationsThe work stands by a standard experimental comparative study which assesses several NLP tools and combining approaches on different test collections and IR models. Thus, it may be helpful for future research works to choose the most suitable tools and develop more sophisticated methods for handling the complexity of Arabic language.Originality/valueThe originality of the idea is to consider that the richness of Arabic is an exploitable characteristic and no more a challenging limit. Thus, the authors combine 4 different morphological levels for the first time in Arabic IR. This approach widely overtook previous research results.Peer reviewThe peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-11-2020-0515

A Review on recent research in information retrieval

In this paper, we present a survey of modeling and simulation approaches to describe information retrieval basics. We investigate its methods, its challenges, its models, its components and its applications. Our contribution is twofold: on the one hand, reviewing the literature on discovery some search techniques that help to get pertinent results and reach an effective search, and on the other hand, discussing the different research perspectives for study and compare more techniques used in information retrieval. This paper will also shedding the light on some of the famous AI applications in the legal field.

THE PREDICTIONS OF PERFORMANCE METRICS IN INFORMATION RETRIEVAL: AN EXPERIMENTAL STUDY

Information retrieval systems are widely used by people from all walks of life to meet diverse user needs. Hence, the ability of these retrieval systems to return the relevant information in response to user queries has been a matter of concern to the information retrieval research community. To address this concern, evaluations of these retrieval systems is extremely critical and the most popular way is the approach that employs test collections. This approach has been the popular evaluation approach in information retrieval for several decades. However, one of the limitations of this evaluation approach concerns the costly creation of relevance judgments. In recent research, this limitation was addressed by predicting performance metrics at the high cut-off depths of documents by using performance metrics computed at low cut-off depths. However, the challenge the research community is faced with is how to predict the precision and the non-cumulative gain performance metrics at the high cut-off depths of documents while using other performance metrics computed at the low cut-off depths of at most 30 documents. This study addresses this challenge by investigating the predictability of performance metrics and proposing two approaches that predict the precision and the non-cumulative discounted gain performance metrics. This study has shown that there exist dataset shifts in the performance metrics computed from different test collections. Furthermore, the proposed approaches have demonstrated better results of the ranked correlations of the predictions of performance metrics than existing research.

A Mathematical Method with Word Frequency Sorting for Mining Web Content Outliers

The Internet is a massive collection of information that makes it extremely difficult to search and retrieve the required and valuable information. So, Search engine became an important tool for searching various data from the web. The primary evaluation of search engine is effectiveness and efficiency. While searching for information through search engines, always users retrieve redundant and irrelevant information. This replicated and uninteresting information affects both the effectiveness and efficiency of search engine by wasting users’ time by browsing the uninterested documents and its accessing time. Web content outlier mining plays a significant role in identifying and removing these redundant document (outliers) which is an important issue among the information retrieval and web mining research communities since most of the people rely on search engines to get the required information. Most existing algorithms for web content outlier mining focuses attention on applying weightage only to the common terms in the documents whereas in this research work, a mathematical approach based on term frequency ranking to identify the duplicates, and uses the domain dictionary to check for relevant document has been carried out to improve the effectiveness and efficiency of the search engine.

Search results diversification for effective fair ranking in academic search

Providing users with relevant search results has been the primary focus of information retrieval research. However, focusing on relevance alone can lead to undesirable side effects. For example, small differences between the relevance scores of documents that are ranked by relevance alone can result in large differences in the exposure that the authors of relevant documents receive, i.e., the likelihood that the documents will be seen by searchers. Therefore, developing fair ranking techniques to try to ensure that search results are not dominated, for example, by certain information sources is of growing interest, to mitigate against such biases. In this work, we argue that generating fair rankings can be cast as a search results diversification problem across a number of assumed fairness groups, where groups can represent the demographics or other characteristics of information sources. In the context of academic search, as in the TREC Fair Ranking Track, which aims to be fair to unknown groups of authors, we evaluate three well-known search results diversification approaches from the literature to generate rankings that are fair to multiple assumed fairness groups, e.g. early-career researchers vs. highly-experienced authors. Our experiments on the 2019 and 2020 TREC datasets show that explicit search results diversification is a viable approach for generating effective rankings that are fair to information sources. In particular, we show that building on xQuAD diversification as a fairness component can result in a significant (p<0.05) increase (up to 50% in our experiments) in the fairness of exposure that authors from unknown protected groups receive.

Report on the 1st IR for children 2000--2020

We present a brief overview of the discussions and takeaways emerging from the IR for Children 2000--2020: Where Are We Now? Workshop, co-located with the 44 th International ACM SIGIR Conference on Research and Development in Information Retrieval. Date : 15 July, 2021. Website : http://www.fab4.science/IR4C/.

Report on the 44th international ACM SIGIR conference on research and development in information retrieval (SIGIR 2021)

ACM SIGIR 2021 conference was organized as a fully online event, with more than 1,000 attendees from dozens of countries, a large technical program, and several activities and initiatives not seen at SIGIR before. This conference report provides some of the important details of how the organizers navigated through changing environments - political, social, and public health - moving the conference from NYC to Montreal and then to online. In addition to that decision process, the report also highlights several new and renewed initiatives around diversity, equity, and inclusion (DEI). Written by the conference General Co-Chairs and the DEI Chairs, this report is meant to inform future conference organizers as well as the SIGIR community at large. Date : 15--21 July, 2021. Website : http://sigir.org/sigir2021/.

Conceptualizing Relevance of Information as a Social Justice Issue: An Interactive Panel Discussion

AbstractRelevance is a notion whose meaning and purpose have been widely discussed in information retrieval research. The ultimate aim of relevance—what Tefko Saracevic has called the “you know” principle—is to ensure that users have the information necessary to meet their goals. What is often missing from this discussion is a critical assessment of who gets to decide what information is relevant, under what circumstances, and for what purposes—especially in relation to marginalized populations. The limited discussion of social justice in information relevance research is a gap this interactive panel discussion seeks to address. Five emerging, junior, and senior researchers will each identify and outline social justice themes of information relevance (e.g., intersectionality, Black feminist lens, geography, immigration status, and holistic critical relevance) that have been evident in their own scholarship. Audience members will have an opportunity to expand on one of the five themes in concert with a panelist before participants consider together future directions for relevance of information in a social justice context. While each participant may have different definitions of social justice, a broad interpretation of the term will frame the conversation by indicating how information relevance can move society towards a fairer and more equitable future.

Authentic versus synthetic: An investigation of the influences of study settings and task configurations on search behaviors

AbstractIn information seeking and retrieval research, researchers often collect data about users' behaviors to predict task characteristics and personalize information for users. The reliability of user behavior may be directly influenced by data collection methods. This article reports on a mixed‐methods study examining the impact of study setting (laboratory setting vs. remote setting) and task authenticity (authentic task vs. simulated task) on users' online browsing and searching behaviors. Thirty‐six undergraduate participants finished one lab session and one remote session in which they completed one authentic and one simulated task. Using log data collected from 144 task sessions, this study demonstrates that the synthetic lab study setting and simulated tasks had significant influences mostly on behaviors related to content pages (e.g., page dwell time, number of pages visited per task). Meanwhile, first‐query behaviors were less affected by study settings or task authenticity than whole‐session behaviors, indicating the reliability of using first‐query behaviors in task prediction. Qualitative interviews reveal why users were influenced. This study addresses methodological limitations in existing research and provides new insights and implications for researchers who collect online user search behavioral data.

Disambiguation of author entities in ADS using supervised learning and graph theory methods

Disambiguation of authors in digital libraries is essential for many tasks, including efficient bibliographical searches and scientometric analyses to the level of individuals. The question of how to link documents written by the same person has been given much attention by academic publishers and information retrieval researchers alike. Usual approaches rely on publications’ metadata such as affiliations, email addresses, co-authors, or scholarly topics. Lack of homogeneity in the structure of bibliographic collections and discipline-specific dissimilarities between them make the creation of general-purpose disambiguators arduous. We present an algorithm to disambiguate authorships in the Astrophysics Data System (ADS) following an established semi-supervised approach of training a classifier on authorship pairs and clustering the resulting graphs. Due to the lack of high-signal features such as email addresses and citations, we engineer additional content- and location-based features via text embeddings and named-entity recognition. We train various nonlinear tree-based classifiers and detect communities from the resulting weighted graphs through label propagation, a fast yet efficient algorithm that requires no tuning. The resulting procedure reaches reasonable complexity and offers possibilities for interpretation. We apply our method to the creation of author entities in a recent ADS snapshot. The algorithm is evaluated on 39 manually-labeled author blocks comprising 9545 authorships from 562 author profiles. Our best approach utilizes the Random Forest classifier and yields a micro- and macro-averaged BCubed mathrm {F}_1 score of 0.95 and 0.87, respectively. We release our code and labeled data publicly to foster the development of further disambiguation procedures for ADS.