GDESA: Greedy Diversity Encoder with Self-attention for Search Results Diversification

Abstract

Search result diversification aims to generate diversified search results so as to meet the various information needs of users. Most of those existing diversification methods greedily select the optimal documents one-by-one comparing with the selected document sequences. Due to the fact that the information utilities of the candidate documents are not independent, a model based on greedy document selection may not produce the global optimal ranking results. To address this issue, some work proposes to model global document interactions regardless of whether a document is selected, which is inconsistent with actual user behavior. In this article, we propose a new supervised diversification framework as an ensemble of global interaction and document selection. Based on a self-attention encoder-decoder structure and an RNN-based document selection component, the model can simultaneously leverage both the global interactions among all the documents and the interactions between the selected sequence and each unselected document. This framework is called Greedy Diversity Encoder with Self-Attention (GDESA). Experimental results show that GDESA outperforms previous methods that rely just on global interactions, and our further analysis demonstrates that using both global interactions and document selection is necessary and beneficial.