Ad-hoc Information Retrieval Research Articles

Information retrieval algorithms and neural ranking models to detect previously fact-checked information

Although in the last decade several fact-checking organizations have emerged to verify misinformation, fake news has continued to proliferate, especially through social media platforms. Even though adopting improved detection strategies is of utmost importance, the fact-checking process could be optimized by verifying whether a claim has been previously fact-checked. Despite some ad-hoc information retrieval approaches having been recently proposed, the utility of modern (neural) retrieval systems have not been investigated yet. In this paper, we consider the standard two-phases retriever-reranker architecture and benchmark different state-of-the-art techniques from the information retrieval and Q&A literature. We design several experiments on a real-world Twitter dataset to analyze the efficiency and the effectiveness of the benchmark approaches. Our results show that combining standard and neural approaches is the most promising research direction to improve retrievers performance and that complex (neural) rerankers might still be efficient in practice since there is no need to process a high number of documents to improve ranking performance.

Document Representation and Query Expansion Models for Blog Recommendation

We explore several different document representation models and two query expansion models for the task of recommending blogs to a user in response to a query. Blog relevance ranking differs from traditional document ranking in ad-hocinformation retrieval in several ways: (1) the unit of output (the blog) is composed of a collection of documents (the blog posts) rather than a single document, (2) the query represents an ongoing and typically multifaceted interest in the topic rather than a passing ad-hoc information need and (3) due to the propensity of spam, splogs, and tangential comments, the blogosphere is particularly challenging to use as a source for high-quality query expansion terms. We address these differences at the document representation level, by comparing retrieval models that view either the blog or its constituent posts as the atomic units of retrieval, and at the query expansion level, by making novel use of the links and anchor text in Wikipedia1 to expand a user's initial query. We develop two complementary models of blog retrieval that perform at comparable levels of precision and recall. We also show consistent and significant improvement across all models using our Wikipedia expansion strategy.

An analysis of evaluation campaigns in ad-hoc medical information retrieval: CLEF eHealth 2013 and 2014

Since its inception in 2013, one of the key contributions of the CLEF eHealth evaluation campaign has been the organization of an ad-hoc information retrieval (IR) benchmarking task. This IR task evaluates systems intended to support laypeople searching for and understanding health information. Each year the task provides registered participants with standard IR test collections consisting of a document collection and topic set. Participants then return retrieval results obtained by their IR systems for each query, which are assessed using a pooling procedure. In this article we focus on CLEF eHealth 2013 and 2014s retrieval task, which saw topics created based on patients’ information needs associated with their medical discharge summaries. We overview the task and datasets created, and the results obtained by participating teams over these two years. We then provide a detailed comparative analysis of the results, and conduct an evaluation of the datasets in the light of these results. This twofold study of the evaluation campaign teaches us about technical aspects of medical IR, such as the effectiveness of query expansion; the quality and characteristics of CLEF eHealth IR datasets, such as their reliability; and how to run an IR evaluation campaign in the medical domain.

Empirical Evaluation of Social and Traditional Search Tools for Adhoc Information Retrieval

The nature of World Wide Web (www) has evolved over the passage of time. Easier and faster availability of Internet has given rise to huge volumes of data available online. Another cause of huge volumes of data is the emergence of online social networks (like Facebook, Twitter, etc.) which has actually changed the role of data consumers to data generators. Increasing popularity of these online social networks has also changed the way different web services used to be used. For example, Facebook messaging has some impact on usage of emails; twitter usage affects (positively or negatively) online newspaper readings. Both of these platforms are heavily used for information searching. In this paper, we evaluate the role of Facebook and Twitter for academic queries and compare the findings with Google search engines to find out if there is a chance that these online social networks will replace Google sooner. A query set selected from the standard AOL dataset is used for experimentation. Academic related queries are selected and classified by expert users. Findings of Google, Facebook and Twitter are compared against these queries using Mean Average Precision (MAP), as a metrics for evaluation. Results conclude that Google has the dominating factor with a better MAP than Facebook and Twitter.

Combining Relevance Language Modeling and Clarity Measure for Extractive Speech Summarization

Extractive speech summarization, which purports to select an indicative set of sentences from a spoken document so as to succinctly represent the most important aspects of the document, has garnered much research over the years. In this paper, we cast extractive speech summarization as an ad-hoc information retrieval (IR) problem and investigate various language modeling (LM) methods for important sentence selection. The main contributions of this paper are four-fold. First, we explore a novel sentence modeling paradigm built on top of the notion of relevance, where the relationship between a candidate summary sentence and a spoken document to be summarized is discovered through different granularities of context for relevance modeling. Second, not only lexical but also topical cues inherent in the spoken document are exploited for sentence modeling. Third, we propose a novel clarity measure for use in important sentence selection, which can help quantify the thematic specificity of each individual sentence that is deemed to be a crucial indicator orthogonal to the relevance measure provided by the LM-based methods. Fourth, in an attempt to lessen summarization performance degradation caused by imperfect speech recognition, we investigate making use of different levels of index features for LM-based sentence modeling, including words, subword-level units, and their combination. Experiments on broadcast news summarization seem to demonstrate the performance merits of our methods when compared to several existing well-developed and/or state-of-the-art methods.

Minimum Rank Error Language Modeling

Statistical language modeling has been successfully developed for speech recognition and information retrieval. The minimum classification error (MCE) training was undertaken to enhance speech recognition performance by minimizing the word error rate. This paper presents a new minimum rank error (MRE) algorithm for n-gram language model training. Rather than speech recognition, the proposed language models are estimated for information retrieval by considering the metric of average precision. However, the maximization of average precision is closely linked to minimizing the rank error or optimizing the order of the ranked documents. Accordingly, this paper calculates the rank error loss function from the misordering pairs of relevant and irrelevant documents in the rank list. The Bayes risk due to the expected rank loss is minimized to develop the Bayesian retrieval rule for ad-hoc information retrieval. Consequently, the discriminative training of language model is performed by integrating discrimination information from individual relevant documents relative to their corresponding irrelevant documents. Experimental results on TREC collections indicate that the proposed MRE language model improves the order of relevant documents, and degrades that of irrelevant documents. The MRE method achieves significantly higher average precision for test queries than the maximum likelihood and the MCE retrieval models.

CLOSING THE VOCABULARY GAP FOR COMPUTING TEXT SIMILARITY AND INFORMATION RETRIEVAL

This paper studies the integration of lexical semantic knowledge in two related semantic computing tasks: ad-hoc information retrieval and computing text similarity. For this purpose, we compare the performance of two algorithms: (i) using semantic relatedness, and (ii) using a conventional extended Boolean model [13] with additional query expansion. For the evaluation, we use two different test collections in the German language especially suitable to study the vocabulary gap problem: (i) GIRT [5] for the information retrieval task, and (ii) a collection of descriptions of professions built to evaluate a system for electronic career guidance in the information retrieval and text similarity tasks. We found that integrating lexical semantic knowledge increases the performance for both tasks. On the GIRT corpus, the performance is improved only for short queries. The performance on the collection of professional descriptions is improved, but crucially depends on the accurate preprocessing of the natural language essays employed as topics.

Improving English and Chinese Ad-Hoc Retrieval: A Tipster Text Phase 3 Project Report

Both English and Chinese ad-hoc information retrieval were investigated in this Tipster 3 project. Part of our objectives is to study the use of various term level and phrasal level evidence to improve retrieval accuracy. For short queries, we studied five term level techniques that together can lead to good improvements over standard ad-hoc 2-stage retrieval for TREC5-8 experiments. For long queries, we studied the use of linguistic phrases to re-rank retrieval lists. Its effect is small but consistently positive. For Chinese IR, we investigated three simple representations for documents and queries: short-words, bigrams and characters. Both approximate short-word segmentation or bigrams, augmented with characters, give highly effective results. Accurate word segmentation appears not crucial for overall result of a query set. Character indexing by itself is not competitive. Additional improvements may be obtained using collection enrichment and combination of retrieval lists. Our PIRCS document-focused retrieval is also shown to have similarity with a simple language model approach to IR.