Research on Web Robot Detection Technology for Concept Drift

Abstract

Web robots have long been significant participants of the internet and the detection is deemed to be essential. Contemporary web robot detection techniques lack the ability to detect when the characteristics of web robots drastically change over time, namely a phenomenon called concept drift. Such change of web robot’s behavior may lead to the deterioration of detection model performance. In order to maintain high detection performance over time, in this paper, we propose a novel web robot detection framework that consists of a model pool, a reinforcement learning algorithm integrating all the models and a concept drift detection module. First, we employ reinforcement learning to integrate a number of detection models that targets different types of web robots, by dynamically adjusting model weights. Then we apply the drift detection method (DDM) to monitor concept drift and identify the need to retrain the model over time. Experiments are conducted using real website datasets under concept drift. The results demonstrate that our model significantly outperforms the state-of-the-art approaches.