Learning to rank and predict: Multi-task learning for ad hoc retrieval and query performance prediction

Abstract

The ad hoc retrieval task aims at ranking relevant documents to a user query such that the most relevant documents are ranked higher compared to less relevant ones. Given the performance of the ad hoc retrieval task can vary across a range of queries, researchers have extensively explored the interrelated task of query performance prediction, which aims at estimating the quality of the search results for a user query without having access to relevance judgments. Traditionally and to-date, the two tasks have been explored as separate tasks where ad hoc retrieval and query performance prediction have been performed in isolation. In this paper, we propose to learn joint tasks that would perform ad hoc retrieval and at the same time predict the quality of the produced rankings. More specifically, we propose a multi-task learning approach, called Multi-task Query Performance Prediction Framework (M-QPPF), which learns document ranking and query performance prediction tasks simultaneously. In M-QPPF, we adopt a shared BERT layer, which is fine-tuned to learn representations for query-document pairs in the embedding space such that the representations effectively encode the cross-interaction between the query and documents. In addition, we include additional yet separate layers to capture task-specific characteristics. We perform comprehensive experiments against state-of-the-art methods in the query performance prediction and ranking tasks over large-scale datasets including the MS MARCO and TREC DL datasets. The improvements measure up to 18.8% on MS MARCO dataset with a Pearson correlation of 0.604, which is superior to the performance of any state-of-the-art baseline in the query performance prediction task.