Abstract
Topic Detection and Tracking technique (TDT) has been commonly used to identify the hot topics from the huge volume of Internet news information and keep up with the hot news. However, traditional topic detection and tracking methods have shown low accuracy and low efficiency. In this paper, a topic detection system driven by big data is built on the Spark platform, which aims at improving the efficiency of news collecting from the Internet and improving the accuracy and efficiency of topic detection and tracking tasks. This system can be easily employed in a distributed architecture and work as a parallelized news collecting and topic detection system. An improved density-based spatial clustering of application with noise (DBSCAN) clustering algorithm based on the time window is proposed to achieve accurate topic detection with the auxiliary advantage of reducing the time complexity. A parallel KNN based topic tracking algorithm is proposed for the topic tracking task. Experiments including comparison with some baseline algorithms and quantitative and qualitative analyses are conducted on pseudo-distributed Spark platform, which demonstrates the effectiveness and efficiency of the parallelized topic detection system.
Highlights
With the rapid paradigm shift of information access, news and information could be provided by online news websites, mainstream media, and individual users as well
Data collection layer is mainly responsible for the extraction and preprocessing of the news data, which serves as the data source of the hot topic detection system
Facing the challenge of high time complexity in the process of text clustering, the proposed model is based on the time windowed density-based spatial clustering of application with noise (DBSCAN) algorithm and big data platform
Summary
With the rapid paradigm shift of information access, news and information could be provided by online news websites, mainstream media, and individual users as well. Facing the challenge of high time complexity in the process of text clustering, the proposed model is based on the time windowed density-based spatial clustering of application with noise (DBSCAN) algorithm and big data platform. In such settings, the time complexity can be reduced from O(n2) to O(n) as analyzed in latter section. The main work of this paper is as below: 1) An improved DBSCAN clustering algorithm based on the time window is proposed, and it adopts an implementation of parallelization to process a huge amount of data stream. The timestamp feature is taken into consideration to reduce the computation cost
Sign-in/Register to view highlights & summary 
Published Version (
Free)
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call