Information Retrieval Research Research Articles

SKYPER: Legal Case Retrieval via Skeleton-aware Hypergraph Embedding in the Hyperbolic Space

Legal Case Retrieval (LCR) plays a significant role in ensuring judicial justice in various legal systems and has recently received increasing attention in Information Retrieval (IR) research. Existing LCR methods typically encode documents into low-dimensional vectors by using either bag of keywords or pretrained language models. However, such approaches generally overlook the interactions among numerous cases and the hierarchy skeleton between event and event type. In this paper, We propose to construct an event-aware knowledge hypergraph for representing legal cases. And we put forward a SKeleton-aware hYPERgraph representation framework named SKYPER, to learn the case embeddings. Concretely, a global skeleton aggregates coarse-grained type information and roughly locates a case. A local skeleton learns the fine-grained information and precisely locates a case. And SKYPER also uses hyperbolic space strengthens the hierarchy between event and event type and avoids vocabulary mismatch. At the retrieval stage, the SKYPER learned case embedding is leveraged for further case similarity matching with a few annotated similarity ranking data. Extensive experiments demonstrate that SKYPER outperforms existing models on the public LCR datasets LeCard and COLIEE20-T1 in both supervised and unsupervised settings. Especially, the retrieval efficiency is improved by a large margin.

Towards bridging the semantic gap between image and text: An empirical approach

Vlogs, Recordings, news, sport coverages are huge sources of multimodal information that do not just limit to text but extend to audio, images and videos. Applications such as summary generation, image/video captioning, multimodal sentiment analysis, cross modal retrieval requires Computer Vision along with Natural Language Processing techniques to extract relevant information. Information from different modalities must be leveraged in order to extract quality content. Hence, reducing the gap between different modalities is of utmost importance. Image to text conversion is an emerging field and employs the use of encoder decoder architecture. Deep CNNs extract the feature of images and sequence to sequence models are used to generate text description. This paper is a contribution towards the growing body of research in multimodal information retrieval. In order to generate the textual description of images, we have performed 5 experiments using the benchmark Flickr8k dataset. In these experiments we have utilized different architectures - simple sequence to sequence model, attention mechanism, transformer-based architecture to name a few. The results have been evaluated using BLEAU score. Results show that the best descriptions are attained by making use of transformer architecture. We have also compared our results with the pretrained visual model vit-gpt2 that incorporates visual transformer.

A multilayer network diffusion-based model for reviewer recommendation

With the rapid growth of manuscript submissions, finding eligible reviewers for every submission has become a heavy task. Recommender systems are powerful tools developed in computer science and information science to deal with this problem. However, most existing approaches resort to text mining techniques to match manuscripts with potential reviewers, which require high-quality textual information to perform well. In this paper, we propose a reviewer recommendation algorithm based on a network diffusion process on a scholar–paper multilayer network, with no requirement for textual information. The network incorporates the relationship of scholar–paper pairs, the collaboration among scholars, and the bibliographic coupling among papers. Experimental results show that our proposed algorithm outperforms other state-of-the-art recommendation methods that use graph random walk and matrix factorization and methods that use machine learning and natural language processing, with improvements of over 7.62% in recall, 5.66% in hit rate, and 47.53% in ranking score. Our work sheds light on the effectiveness of multilayer network diffusion-based methods in the reviewer recommendation problem, which will help to facilitate the peer-review process and promote information retrieval research in other practical scenes.

Report on the 46th ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2023): Reflections from the Program Co-Chairs

The ACM SIGIR Conference on Research and Development in Information Retrieval has been experiencing significant growth over the past few years. In 2023, SIGIR received a total of 822 full paper submissions and 1,787 papers/proposals. This has brought several challenges to the conference program organization. This article reports some of the main aspects of the work conducted by the program committee (PC) co-chairs for SIGIR 2023, and discusses emergent issues of the review process and program design. We expect that this document serve as a reference for future PC co-chars and as a starting point for further discussions towards improving the review process and overall conference experience. Date : 23--27 July 2023. Website : https://sigir.org/sigir2023/.

Report on the 1st Workshop on Implicit Author Characterization from Texts for Search and Retrieval (IACT 2023) at SIGIR 2023

The first edition of the International Workshop on Implicit Author Characterization from Texts for Search and Retrieval (IACT'23) was held on July 27 th , 2023, in conjunction with the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) in Taipei, Taiwan. To support both online and in-person participation, the workshop was held as a hybrid event. Online participation was allowed using the EventX platform. During the course of the day, about 15 attendees had the opportunity to follow up and discuss the recent advances in topics related to identifying and extracting implicit information about authors (e.g., human or AI) from texts and using it in IR tasks. The workshop program included two invited keynotes, four paper presentations, and the report of the ColiE competition which was held as a part of the workshop1. The proceedings of the workshop are available online at https://ceur-ws.org/Vol-3477/. Date : 27 July 2023. Website : https://en.sce.ac.il/news/iact23.

ACM SIGIR Annual Business Meeting 2023: Secretary's Notes

The 2023 ACM SIGIR Annual Business Meeting took place as a hybrid event during the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval in Taipei on July 26, 2023. The business meeting was led by the chair of the SIGIR Executive Committee (EC), Vanessa Murdock.

Report on the 1st Workshop on Generative Information Retrieval (Gen-IR 2023) at SIGIR 2023

The first edition of the workshop on Generative Information Retrieval (Gen-IR 2023) took place in July 2023 in a hybrid fashion, co-located with the ACM SIGIR Conference 2023 in Taipei (SIGIR 2023). The aim was to bring information retrieval researchers together around the topic of generative AI that gathered attention in 2022 and 2023 with large language models and diffusion models. Given the novelty of the topic, the workshop was focused around multi-sided discussions, namely panels and poster sessions of the accepted proceedings papers. Two main research outcomes are the proceedings of the workshop1 and the potential research directions discussed in this report. Date : 27 July 2023. Website : https://coda.io/@sigir/gen-ir.

Fintech Key-Phrase: A New Chinese Financial High-Tech Dataset Accelerating Expression-Level Information Retrieval

Expression-level information extraction is a challenging task in natural language processing (NLP), which aims to retrieve crucial semantic information from linguistic documents. However, there is a lack of up-to-date data resources for accelerating expression-level information extraction, particularly in the Chinese financial high technology field. To address this gap, we introduce Fintech Key-Phrase: a human-annotated key-phrase dataset for the Chinese financial high technology domain. This dataset comprises over 12K paragraphs along with annotated domain-specific key-phrases. We extract the publicly released reports, Chinese management’s discussion and analysis (CMD&A), from the renowned Chinese research data services platform (CNRDS) and then filter the reports related to high technology. The high technology key-phrases are annotated following pre-defined philosophy guidelines to ensure annotation quality. In order to better understand the limitations and challenges in the purposed dataset, we conducted comprehensive noise evaluation experiments for the Fintech Key-Phrase, including annotation consistency assessment and absolute annotation quality evaluation. To demonstrate the usefulness of our released Fintech Key-Phrase in retrieving valuable information in the Chinese financial high technology field, we evaluate its significance using several superior information retrieval systems as representative baselines and report corresponding performance statistics. Additionally, we further applied ChatGPT to the text augmentation approach of the Fintech Key-Phrase dataset. Extensive comparative experiments demonstrate that the augmented Fintech Key-Phrase dataset significantly improved the coverage and accuracy of extracting key phrases in the finance and high-tech domains. We believe that this dataset can facilitate scientific research and exploration in the Chinese financial high technology field. We have made the Fintech Key-Phrase dataset and the experimental code of the adopted baselines accessible on Github: https://github.com/albert-jin/Fintech-Key-Phrase. To encourage newcomers to participate in the financial high-tech domain information retrieval research, we have developed a series of tools, including an open website1and corresponding real-time information retrieval APIs.2

High-Quality and Reproducible Automatic Drum Transcription from Crowdsourced Data

Within the broad problem known as automatic music transcription, we considered the specific task of automatic drum transcription (ADT). This is a complex task that has recently shown significant advances thanks to deep learning (DL) techniques. Most notably, massive amounts of labeled data obtained from crowds of annotators have made it possible to implement large-scale supervised learning architectures for ADT. In this study, we explored the untapped potential of these new datasets by addressing three key points: First, we reviewed recent trends in DL architectures and focused on two techniques, self-attention mechanisms and tatum-synchronous convolutions. Then, to mitigate the noise and bias that are inherent in crowdsourced data, we extended the training data with additional annotations. Finally, to quantify the potential of the data, we compared many training scenarios by combining up to six different datasets, including zero-shot evaluations. Our findings revealed that crowdsourced datasets outperform previously utilized datasets, and regardless of the DL architecture employed, they are sufficient in size and quality to train accurate models. By fully exploiting this data source, our models produced high-quality drum transcriptions, achieving state-of-the-art results. Thanks to this accuracy, our work can be more successfully used by musicians (e.g., to learn new musical pieces by reading, or to convert their performances to MIDI) and researchers in music information retrieval (e.g., to retrieve information from the notes instead of audio, such as the rhythm or structure of a piece).

Conditional variational autoencoder for query expansion in ad-hoc information retrieval

Query expansion (QE) is commonly used to improve the performance of traditional information retrieval (IR) models. With the adoption of deep learning in IR research, neural QE models have emerged in recent years. Many of these models focus on learning embeddings by leveraging query-document relevance. These embedding models allow computing semantic similarities between queries and documents to generate expansion terms. However, existing models often ignore query-document interactions. This research aims to address that gap by proposing a QE model using a conditional variational autoencoder. It first maps a query-document pair into a latent space based on their interaction, then estimates an expansion model from that latent space. The proposed model is trained on relevance feedback data and generates expansions using pseudo-relevance feedback at test time. The proposed model is evaluated on three standard TREC collections for document ranking: AP and Robust 04 and GOV02, and the MS MARCO dataset for passage ranking. Results show the model outperforms state-of-the-art traditional and neural QE models. It also demonstrates higher additivity with neural matching than baselines.

Perceptual and automated estimates of infringement in 40 music copyright cases

Music copyright infringement lawsuits implicate millions of dollars in damages and costs of litigation. There are, however, few objective measures by which to evaluate these claims. Recent music information retrieval research has proposed objective algorithms to automatically detect musical similarity, which might reduce subjectivity in music copyright infringement decisions, but there remains minimal relevant perceptual data despite its crucial role in copyright law. We collected perceptual data from 51 participants for 40 adjudicated copyright cases from 1915–2018 in 7 legal jurisdictions (USA, UK, Australia, New Zealand, Japan, People’s Republic of China, and Taiwan). Each case was represented by three different versions: either full audio, melody only (MIDI), or lyrics only (text). Due to the historical emphasis in legal opinions on melody as the key criterion for deciding infringement, we originally predicted that listening to melody-only versions would result in perceptual judgments that more closely matched actual past legal decisions. However, as in our preliminary study of 17 court decisions (Yuan et al., 2020), our results did not match these predictions. Participants listening to full audio outperformed not only the melody-only condition, but also automated algorithms designed to calculate musical similarity (with maximal accuracy of 83% vs. 75%, respectively). Meanwhile, lyrics-only conditions performed at chance levels. Analysis of outlier cases suggests that music, lyrics, and contextual factors can interact in complex ways difficult to capture using quantitative metrics. We propose directions for further investigation including using larger and more diverse samples of cases, enhanced methods, and adapting our perceptual experiment method to avoid relying on ground truth data only from court decisions (which may be subject to errors and selection bias). Our results contribute data and methods to inform practical debates relevant to music copyright law throughout the world, such as the question of whether, and the extent to which, judges and jurors should be allowed to hear published sound recordings of the disputed works in determining musical similarity. Our results ultimately suggest that while automated algorithms are unlikely to replace human judgments, they may help to supplement them.

Beyond Comparative Analysis: Making Arguments with Similarity Metrics and Structured Manuscript Data, with a Case Study in Marginal Iconography

Abstract: Quantitative measurements of similarity can help researchers understand the increasing amount of manuscript metadata or digital images now available for study. Researchers in archaeology, ecology, and information retrieval have developed diverse methods for quantifying the similarity of categorical, numerical, and presence/absence data. The variety of methods in use reflects the complex associations of the term "similarity" and suggests that quantitative approaches can preserve some of the nuance associated with traditional humanistic approaches. We argue that similarity metrics, contextualized against random simulations, provide flexible tools for identifying and analyzing trends in large sets of multifaceted historical data and can be applied to many kinds of manuscript research projects. Simulations (permutation tests) that highlight how observed patterns diverge from random or theoretical scenarios have allowed us to interpret similarity measurements against a baseline that is analogous to researchers' intuitions born of experience. In this paper, we model using similarity metrics to conduct multifaceted analyses of an iconographic dataset derived from Lilian M. C. Randall's Images in the Margins of Gothic Manuscripts (1966), indexing images in French, English, and Flemish manuscripts produced between about 1250 and 1350. We share examples of approaches such as statistical analysis, Analysis of Similarity, and clustering to model the type of quantitative approaches we believe will become increasingly important for the analysis of digital manuscript data.

A Systematic Review of Cost, Effort, and Load Research in Information Search and Retrieval, 1972–2020

During the information search and retrieval (ISR) process, user-system interactions such as submitting queries, examining results, and engaging with information impose some degree of demand on the user’s resources. Within ISR, these demands are well recognised, and numerous studies have demonstrated that the cost, effort, and load (CEL) experienced during the search process are affected by a variety of factors. Despite this recognition, there is no universally accepted definition of the constructs of CEL within the field of ISR. Ultimately, this has led to problems with how these constructs have been interpreted and subsequently measured. This systematic review contributes a synthesis of literature, summarising key findings relating to how researchers have been defining and measuring CEL within ISR over the past 50 years. After manually screening 1,109 articles, we detailed and analysed 91 articles which examine CEL within ISR. The discussion focuses on comparing the similarities and differences between CEL definitions and measures before identifying the limitations of the current state of the nomenclature. Opportunities for future research are also identified. Going forward, we propose a CEL taxonomy that integrates the relationships between CEL and their related constructs, which will help focus and disambiguate future research in this important area.

Computational approaches to detect experts in distributed online communities: a case study on Reddit

AbstractThe irreplaceable key to the triumph of Question & Answer (Q & A) platforms is their users providing high-quality answers to the challenging questions posted across various topics of interest. From more than a decade, the expert finding problem attracted much attention in information retrieval research. Based on the encountered gaps in the expert identification across several Q & A portals, we inspect the feasibility of identifying data science experts in Reddit. Our method is based on the manual coding results where two data science experts labelled not only expert and non-expert comments, but also out-of-scope comments, which is a novel contribution to the literature, enabling the identification of more groups of comments across web portals. We present a semi-supervised approach which combines 1113 labelled comments with 100,226 unlabelled comments during training. We proved that it is possible to develop models that can identify expert, non-expert and out-of-scope comments peaking the AUC score at 0.93, accuracy at 0.83, MAE at 0.15 degrees and R2 score at 0.69. The proposed model uses the activity behaviour of every user, including Natural Language Processing (NLP), crowdsourced and user feature sets. We conclude that the NLP and user feature sets contribute the most to the better identification of these three classes. It means that this method can generalise well within the domain. Finally, we make a novel contribution by presenting different types of users in Reddit, which opens many future research directions.

Deep Chnet 1-D CNN using raw audio waveform for recognizing traditional Chinese musical instrument

The classification of music information using various deep learning models is increasingly popular in the field of Music Information Retrieval research. However, as most proposed works focus on western music and musical instruments, little attention is given to traditional Chinese music. This paper proposes a 1-D Convolutional Neural Network (1-D CNN) using only raw audio waveform as input, to undertake the task of traditional Chinese musical instruments classification. This paper starts with a review of the current state of research on the related field, then discuss the proposed model and its data in detail, followed by its performance metrics and then a conclusion on the experiment. The result shows that 1-D CNN provides competitive and even superior results when compared to its 2-D versions as well as when compared to traditional models.

Persistence, vision, networks, collaboration - leadership in information retrieval research

The library and information science community has its share of inspirational research pioneers building a corpus of knowledge to influence and construct the future. Intuitive search interface design, the psychology and mechanics of information retrieval and information seeking behaviour are key development priorities. In December 2021, Ian Ruthven, Professor of Information Seeking and Retrieval at the Department of Computer and Information Sciences, University of Strathclyde, presented the seventh annual Strix Memorial Lecture entitled: ‘Google’s what you use when Alexa doesn’t know the answer, Uncle Ian.’ It was inspired by an innocent comment made by his young niece. People create their own models of how information works for them. Information seeking behaviour is social and shaped by life, so there are infinite iterations of queries and limitless perceived solutions to perceived problems. We explore Ian’s work and his observation that there is so much potential to progress research by placing information seeking and retrieval within the broader societal and cultural framework of what people or populations want from or do with information.

ASPER: Attention-based approach to extract syntactic patterns denoting semantic relations in sentential context

Semantic relationships, such as hyponym–hypernym, cause–effect, meronym–holonym etc., between a pair of entities in a sentence are usually reflected through syntactic patterns. Automatic extraction of such patterns benefits several downstream tasks, including, entity extraction, ontology building, and question answering. Unfortunately, automatic extraction of such patterns has not yet received much attention from NLP and information retrieval researchers. In this work, we propose an attention-based supervised deep learning model, ASPER, which extracts syntactic patterns between entities exhibiting a given semantic relation in the sentential context. We validate the performance of ASPER on three distinct semantic relations—hyponym–hypernym, cause–effect, and meronym–holonym on six datasets. Experimental results show that for all these semantic relations, ASPER can automatically identify a collection of syntactic patterns reflecting the existence of such a relation between a pair of entities in a sentence. In comparison to the existing methodologies of syntactic pattern extraction, ASPER’s performance is substantially superior.

Combining variable neighborhood with gradient ascent for learning to rank problem

Metaheuristic applications for information retrieval research are limited in spite of the importance of this problem domain. Ranking the retrieved documents based on their importance is a vital issue for the scientific and industrial communities. This paper proposes a novel variable neighborhood search (VNS) algorithm with adaptation based on an objective function for the learning to rank (LTR) problem. VNS is a global optimum metaheuristic algorithm that has been engaged to evolve the optimal solutions for heuristic problems based on exploring better neighbor solutions from the current one. The changes from the current to the next optimal solution are made during the perturbation stage to identify the global optimal solutions. The exploration procedure has been made through various mutation step sizes, whereas the exploitation process has been done by checking the quality of the evolved solutions using the fitness function. This research proposes a novel version of VNS based on four random probability distributions with gradient ascent. In addition to using the traditional random generator with gradient ascent for modifying the mutated genes of the neighborhood candidate solution in the following evolving iteration. This novel method in LTR is called gradient variable neighborhood (GVN). In the experiments, we utilized Microsoft Bing search (MSLR-WEB30K), Yahoo, and TREC Million Queries Competitions in 2008 and 2007 (LETOR 4) datasets. From the findings of the results, we can deduce that the GVN method outperformed recent studies on LTR methods.

Information Retrieval meets Large Language Models: A strategic report from Chinese IR community

The research field of Information Retrieval (IR) has evolved significantly, expanding beyond traditional search to meet diverse user information needs. Recently, Large Language Models (LLMs) have demonstrated exceptional capabilities in text understanding, generation, and knowledge inference, opening up exciting avenues for IR research. LLMs not only facilitate generative retrieval but also offer improved solutions for user understanding, model evaluation, and user-system interactions. More importantly, the synergistic relationship among IR models, LLMs, and humans forms a new technical paradigm that is more powerful for information seeking. IR models provide real-time and relevant information, LLMs contribute internal knowledge, and humans play a central role of demanders and evaluators to the reliability of information services. Nevertheless, significant challenges exist, including computational costs, credibility concerns, domain-specific limitations, and ethical considerations. To thoroughly discuss the transformative impact of LLMs on IR research, the Chinese IR community conducted a strategic workshop in April 2023, yielding valuable insights. This paper provides a summary of the workshop’s outcomes, including the rethinking of IR’s core values, the mutual enhancement of LLMs and IR, the proposal of a novel IR technical paradigm, and open challenges.

Understanding Feature Importance in Musical Works: Unpacking Predictive Contributions to Cluster Analyses

Cluster analysis provides insight into musical patterns in composition, performance, and perception. Despite its wide adoption in music research, understanding how specific features affect clustering solutions remains challenging. For example, features such as mode (i.e., major/minor), timing, signal amplitude, and pitch are often intercorrelated, making it difficult to understand their specific role within different clusters. To demonstrate how accumulated local effects (ALEs) can help with this challenge, here we analyze 48 excerpts from complete sets of preludes by Bach and Chopin, showing how specific features contribute to two- and three-cluster analyses. These exploratory analyses reveal that ALEs can identify salient or subtle data patterns from cluster analyses by tracking how changes in features affect cluster membership. We explore these insights in visualizations quantifying feature importance and an interactive companion application ( https://maplelab.net/feature-importance/ ) featuring the analyzed audio. Following a demonstration of this method, we suggest how it can be applied to explore topics of interest to researchers in music information retrieval, empirical musicology, and music cognition alike.