Sem-TED: Semantic Twitter Event Detection and Adapting with News Stories

Abstract

Public acceptance of social networks has made the analysis of these networks essential. Event detection in these networks including Twitter is one of the most momentous subjects in the field of natural language processing and text mining. In this paper, we investigated how to link popular social media topics and news stories using transformer models and neural networks. Accordingly, this study consists of two parts: First, detecting popular topics and, second, linking them to the news. Event detection techniques have been applied to detect popular topics, while an event detection method comprises text preprocessing, text embedding using Sentence Transformer, dimension reduction using the UMAP algorithm, and grouping them using the HDBSCAN algorithm. To examine relevance or non-relevance between the news and topics, a single-layer perceptron neural network is applied, in which the output of the model indicates relevance or nonrelevance. We have implemented the mentioned parts and have investigated them on a small sampling of two known datasets. The evaluation outcomes reveal that the first part leads to an average improvement of 8% compared to the entity-based methods. Moreover, the results of the second part demonstrate that the used neural network in this study has a better performance comparing several other methods.