Challenges and design issues of an Arabic web crawler

Abstract

The world-wide-web has become the favorite destiny of information seekers across the globe. With its massive amount of information that includes billions of web pages, information for just about any topic is a click-of-finger away. The Arabic web represents an important portion of the web. With Arabic as the 5th most spoken language in the world and with the increasing number of Arabic internet users at exponential rates, it is becoming important to make the Arabic web content available. Many search engines exist such as Google, Msn, and Yahoo. However, these search engines are not designed with proper consideration of the Arabic language and its special nature. This paper presents some of the work while developing an Arabic search engine. The web crawler is the focus of this paper. We present important challenges and design issues of the design of a distributed web crawler for the Arabic search engine we are developing. We also uncover some of the facts about the Arabic web.