A set of novel HTML document quality features for Web information retrieval: Including applications to learning to rank for information retrieval

Abstract

The past work on Information Retrieval (IR) targeting web document collections shows that incorporating a measure that measures the quality of web documents, or rather the document prior (e.g., PageRank), into an IR system improves the retrieval effectiveness. In this study, we introduce new document priors and empirically investigate their effect by employing them as features in a learning to rank (LTR) deployment. The experiments are performed on the two standard Web IR test collections: the ClueWeb09 and the ClueWeb12 datasets, which include 500 and 733 million web documents, respectively, and the associated TREC & NTCIR query sets with a total number of 1,204 queries. A strong baseline is formed by using standard features introduced in the previous works, with respect to which the effect of newly introduced features in this paper is empirically compared. We test our features by LambdaMART, which is state-of-the-art LTR technique. The results reveal that the features introduced in this work led improvement in retrieval performance on the test collections in use. The introduced features are classified into 5 groups with respect to functional properties and each group is also analyzed in detail.