Web bots that mimic human browsing behavior on previously unvisited web-sites: Feasibility study and security implications

Abstract

In the past, there have been many attempts at developing accurate models of human-like browsing behavior. However, most of these attempts/models suffer from one of following drawbacks: they either require that some previous history of actual human browsing on the target web-site be available (which often is not the case); or, they assume that ‘think times’ and ‘page popularities’ follow the well-known Poisson and Zipf distribution (an old hypothesis that does not hold well in the modern-day WWW). To our knowledge, our work is the first attempt at developing a model of human-like browsing behavior that requires no prior knowledge or assumption about human behavior on the target site. The model is founded on a more general theory that defines human behavior as an ‘interest-driven’ process. The preliminary simulation results are very encouraging - web bots built using our model are capable of mimicking real human browsing behavior 1000-fold better compared to bots that deploy random crawling strategy.