Design and Realization of SVM Topic Crawler Based on Incremental Learning

Abstract

Using information technology such as search engine for collecting and monitoring of network public opinion is a practical and effective method. This paper puts forward an improved algorithm of SVM classifier based on incremental learning online, and then implements a topic crawler system for network public opinion and to grab the public opinion. SVM (Support Vector Machine) is first proposed by Vapnik in 1995. It exhibits many unique advantages in addressing the small sample, nonlinear and high dimensional pattern recognition. SVM pattern recognition is based on statistical learning theory method has important applications in computer pattern recognition field. The SVM research is not perfect, cannot efficiently solve pattern recognition problems. But with the study of the application of statistical learning theory and neural networks than the new machine learning methods encounter some significant difficulties, such as how to determine the issue of network structure, through learning and learning problems due to local minima problems, leading to many researchers added to the study of SVM classification algorithm to improve. This effectively promotes the rapid development of SVM classification algorithm and continuous improvement, and SVM classifier was quickly extended to other machine learning problems fitting in function today SVM classification algorithm on text categorization has been successful applications. Since the 1990s, the rapid development of Internet technology, automatic text classification research has entered a new stage, based on machine learning text classification technology gradually replaced the method based on knowledge engineering automatic text classification has become the main form. Carried out in comparison with the classification Bayesian k-nearest neighbor and decision tree, the support vector machine method achieved the best classification accuracy since more and more researchers began to pay attention to them, and for the support of two standard corpus SVM and text classification were studied and put forward a number of new methods. In recent years, the introduction of SVM classifier topic crawlers, used to guide and supervise the theme crawler got the attention of many scholars. Johnson first SVM classification algorithm supervision focused crawler conducted theoretical research, proposed a SVM classifier model to guide the crawling reptile theme, and a lot of related experiments. More and more scholars began to use support vector machine guidance and supervision topic reptiles, Michelangelo and other support vector machine to guide the theme reptiles, and proposes a support vector machine classification algorithm page. Topic relevance determination method of obtaining a direct impact on the rate of theme crawler, traditional themes reptiles crawling in the website, acquisition rate has been low. Based on SVM classifier, this paper further studies SVM classification algorithm in the prediction of the page subject classification and proposes an incremental learning of SVM classification algorithm, and finally applies it to the network crawl on public opinion.