The Research on the Spider of the Domain-Specific Search Engines Based on the Reinforcement Learning

Abstract

On the basis of the comparison between general search engines and vertical search engines, the development of vertical search engines was discussed. After the introduction of the theory and algorithms of reinforcement learning, the reinforcement learning methods was applied to the spider, a vertical search engine, as a new technology. This developed the application of reinforcement learning and provide a new idea for spider. From the experimental results, the conclusion is that the performance improved a lot with this technology in spider search by the experimental analysis.