Using Dark Web Crawler to Uncover Suspicious and Malicious Websites

Abstract

It has been recognized that most of the Internet is not accessible through regular search engines and web browsers. This part of the web is known as dark web, and the surface is about 400 to 500 times larger than the size of the web that we know [1]. The goal of this project is to design a dark web crawler that can uncover any suspicious and malicious websites from the TOR (The Onion Router) network. The proposed system will create a database of suspicious and malicious websites by scraping relative linking attributes that may be contained within TOR network web pages. The proposed database automatically updates itself and it will archive previous versions of TOR sites while saving available working links. This will give law enforcement authorities the ability to search both the current TOR database and previous versions of the Database to detect suspicious and malicious websites.