Analysis on the content features and their correlation of web pages for spam detection

Abstract

In the global information era, people acquire more and more information from the Internet, but the quality of the search results is degraded strongly because of the presence of web spam. Web spam is one of the serious problems for search engines, and many methods have been proposed for spam detection. We exploit the content features of non-spam in contrast to those of spam. The content features for non-spam pages always possess lots of statistical regularities; but those for spam pages possess very few statistical regularities, because spam pages are made randomly in order to increase the page rank. In this paper, we summarize the regularities distributions of content features for non-spam pages, and propose the calculating probability formulae of the entropy and independent n-grams respectively. Furthermore, we put forward the calculation formulae of multi features correlation. Among them, the notable content features may be used as auxiliary information for spam detection.