Abstract

Web page de-duplication module is an important part of search engine system, which can improve its performance and quality with filtering the web pages downloaded by crawler system of search engine and eliminating the duplicated web pages. This paper from the source of duplicated web pages--reshipment proposes a web page de-duplication method that the information including original websites and web titles are extracted to eliminate duplicated web pages based on feature codes. Experiments show that this method can achieve satisfactory results in eliminating large-scale duplicated web pages.

Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call