Using gradient descent to optimize language modeling smoothing parameters

Abstract

Most information retrieval models have some set of tunable parameters. It is often the case that the parameters of such models are either set to default values or tuned in a supervised or unsupervised manner. Historically, many retrieval models have been tuned to maximize some underlying retrieval metric, such as mean average precision. However, as the number of parameters increases, the direct maximization techniques become computationally expensive. For this reason, there has been a growing interest in both the information retrieval and machine learning communities to develop parameter estimation techniques that scale well. Since most information retrieval metrics are non-smooth with respect to model parameters, the machine learning techniques have focused on maximizing or minimizing some surrogate function that attempts to mimic or be highly correlated with retrieval metrics. Standard optimization techniques can then easily be applied to the surrogate function in order to estimate approximate parameters. There have been, however, few studies from an information retrieval perspective into how well such surrogate functions compare to the direct search approach. Recent work has investigated how the effectiveness of using BM25F parameters estimated by minimizing the RankNet cost function correlates with the effectiveness of an approach that directly maximizes NDCG [2]. The results showed that RankNet acts as a good surrogate and produces reasonable effectiveness when compared to a more computationally expensive direct search technique. Following up on this work, we wish to explore how to use the RankNet cost function to optimize language modeling smoothing parameters. In addition, we explore how well the parameters learned using RankNet compare to those found by a direct search technique for various metrics that rely on binary relevance judgments, such as mean average precision, binary preference, and precision at 10.