Abstract
Web crawler technology is the technology which downloads web pages through the program. This paper implements incremental Python web crawler, uses Scrapy crawler framework, crawls news web pages from mainstream web sites incrementally in real time, and deposits data in the database. The key technology of incremental crawling is removing the repetition of web links, and the most common method is using Bloom filter. This paper implements a simplified Bloom filter. The result shows that the web crawler can monitor the news pages well.
Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
